[CARBONDATA-1218] [GLOBAL SORT] In case of data-load failure the BadRecordsLogger.addRecordEntry map holding the task Status is not removing the task Entry. #1082

mohammadshahidkhan · 2017-06-23T09:27:33Z

Problem
For GLOBAL_SORT scope option in case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry.
Because of this the next load is getting failed even though the data being loaded has no bad records.
Solution
The map entry must be removed after load completion either success or fail.
Refactored the Bad record logger.

Any interfaces changed?
NA
Any backward compatibility impacted?
NA
Document update required?
NA
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
Added Test case
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

asfgit · 2017-06-23T09:27:35Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-06-23T09:38:30Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/98/

CarbonDataQA · 2017-06-23T09:39:05Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2670/

asfgit · 2017-06-23T10:13:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/591/

Failed Tests: 4

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 4

org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option to load command
org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option to load command and compare with normal load
org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option and compaction
org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option in one load and with out option in other load and then do compaction

asfgit · 2017-06-28T07:40:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/691/

CarbonDataQA · 2017-06-28T07:42:55Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/183/

CarbonDataQA · 2017-06-28T07:46:11Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2762/

CarbonDataQA · 2017-06-29T16:54:30Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/232/

CarbonDataQA · 2017-06-29T16:55:58Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2812/

CarbonDataQA · 2017-06-30T05:49:06Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2822/

CarbonDataQA · 2017-06-30T05:50:59Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/242/

CarbonDataQA · 2017-06-30T06:03:16Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2823/

CarbonDataQA · 2017-06-30T06:04:41Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/243/

CarbonDataQA · 2017-08-02T08:46:24Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-09-13T02:33:04Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/52/

ravipesala · 2017-10-23T13:07:08Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1228/

CarbonDataQA · 2017-10-23T17:16:50Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/597/

CarbonDataQA · 2017-12-16T18:14:37Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/832/

CarbonDataQA · 2017-12-16T18:30:49Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2059/

CarbonDataQA · 2017-12-16T18:44:47Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/837/

ravipesala · 2017-12-16T19:20:24Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2368/

CarbonDataQA · 2017-12-16T22:30:25Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2063/

mohammadshahidkhan · 2017-12-18T09:46:54Z

retest this please

CarbonDataQA · 2017-12-18T12:53:49Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/871/

CarbonDataQA · 2017-12-18T13:02:27Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2096/

CarbonDataQA · 2017-12-19T16:38:00Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/914/

ravipesala · 2017-12-19T18:58:16Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2427/

CarbonDataQA · 2017-12-19T20:20:12Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2144/

mohammadshahidkhan · 2017-12-21T06:06:10Z

retest this please

ravipesala · 2017-12-21T07:04:27Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2471/

CarbonDataQA · 2018-01-02T07:28:50Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2470/

CarbonDataQA · 2018-01-02T07:31:42Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1246/

CarbonDataQA · 2018-01-02T08:36:40Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2473/

ravipesala · 2018-01-02T08:58:06Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2650/

CarbonDataQA · 2018-01-02T10:03:46Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1249/

ravipesala · 2018-01-02T11:48:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2653/

manishgupta88 · 2018-01-04T05:50:45Z

...rc/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordLoggerTest.scala

+    assert(true)
+    sql("drop table IF EXISTS loadIssue")
+  }
+


Use only one try and finally block inside test code..try should start from first line and end at last and finally should only reset the carbon property

Instead of try catch use intercept[Exception] only for the 1st case where exception is expected. For 2nd case try catch is not required...if exception comes anyways test case will fail

manishgupta88 · 2018-01-04T06:22:35Z

processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java

+    // rename the bad record in progress to normal
+    CarbonTableIdentifier identifier =
+        configuration.getTableIdentifier().getCarbonTableIdentifier();
+    CarbonDataProcessorUtil.renameBadRecordsFromInProgressToNormal(configuration,


Avoid calling the method using classname. Method is present in the same class.

I think we can introduce a new class called BadRecordsUtil which will have all the functions related to bad records like
renameBadRecord, hasBadRecord, renameBadRecordsFromInProgressToNormal (make it private in the new util class)

…adRecordEntry map holding the task Status is not removing the task Entry

CarbonDataQA · 2018-01-04T17:07:35Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2554/

ravipesala · 2018-01-04T22:04:11Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2720/

CarbonDataQA · 2018-01-05T00:11:16Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1331/

CarbonDataQA · 2018-01-05T00:28:52Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2567/

ravipesala · 2018-01-05T04:33:08Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2731/

…adRecordEntry map holding the task Status is not removing the task Entry Problem For GLOBAL_SORT scope option in case of data-load failure the BadRecordsLogger.badRecordEntry map holding the task Status is not removing the task Entry. Because of this the next load is getting failed even though the data being loaded has no bad records. Solution The map entry must be removed after load completion either success or fail. Refactored the Bad record logger. This closes apache#1082

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 7106938 to cfbe473 Compare June 28, 2017 07:21

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 436097b to 88102db Compare June 30, 2017 05:46

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 88102db to 944455a Compare June 30, 2017 05:51

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 944455a to 7e1a68f Compare December 16, 2017 17:25

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 7e1a68f to 8ea1148 Compare December 16, 2017 18:08

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 8ea1148 to b1c938e Compare December 19, 2017 13:26

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from b1c938e to 5c5202f Compare December 21, 2017 06:05

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch 2 times, most recently from 6b95d4a to 81741eb Compare January 2, 2018 07:03

manishgupta88 suggested changes Jan 4, 2018

View reviewed changes

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from 81741eb to df6e7c2 Compare January 4, 2018 10:41

[CARBONDATA-1218] In case of data-load failure the BadRecordsLogger.b…

5010de3

…adRecordEntry map holding the task Status is not removing the task Entry

mohammadshahidkhan force-pushed the task_status_cleanup_problem branch from f8e56f9 to 5010de3 Compare January 4, 2018 16:46

asfgit closed this in 837fdd2 Jan 5, 2018

[CARBONDATA-1218] [GLOBAL SORT] In case of data-load failure the BadRecordsLogger.addRecordEntry map holding the task Status is not removing the task Entry. #1082

[CARBONDATA-1218] [GLOBAL SORT] In case of data-load failure the BadRecordsLogger.addRecordEntry map holding the task Status is not removing the task Entry. #1082

Conversation

mohammadshahidkhan commented Jun 23, 2017 • edited

asfgit commented Jun 23, 2017

CarbonDataQA commented Jun 23, 2017

CarbonDataQA commented Jun 23, 2017

asfgit commented Jun 23, 2017

Failed Tests: 4

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 4

asfgit commented Jun 28, 2017

CarbonDataQA commented Jun 28, 2017

CarbonDataQA commented Jun 28, 2017

CarbonDataQA commented Jun 29, 2017

CarbonDataQA commented Jun 29, 2017

CarbonDataQA commented Jun 30, 2017

CarbonDataQA commented Jun 30, 2017

CarbonDataQA commented Jun 30, 2017

CarbonDataQA commented Jun 30, 2017

CarbonDataQA commented Aug 2, 2017

CarbonDataQA commented Sep 13, 2017

ravipesala commented Oct 23, 2017

CarbonDataQA commented Oct 23, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

ravipesala commented Dec 16, 2017

CarbonDataQA commented Dec 16, 2017

mohammadshahidkhan commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 18, 2017

CarbonDataQA commented Dec 19, 2017

ravipesala commented Dec 19, 2017

CarbonDataQA commented Dec 19, 2017

mohammadshahidkhan commented Dec 21, 2017

ravipesala commented Dec 21, 2017

CarbonDataQA commented Jan 2, 2018

CarbonDataQA commented Jan 2, 2018

CarbonDataQA commented Jan 2, 2018

ravipesala commented Jan 2, 2018

CarbonDataQA commented Jan 2, 2018

ravipesala commented Jan 2, 2018

manishgupta88 Jan 4, 2018

Choose a reason for hiding this comment

manishgupta88 Jan 4, 2018

Choose a reason for hiding this comment

mohammadshahidkhan Jan 4, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jan 4, 2018

ravipesala commented Jan 4, 2018

CarbonDataQA commented Jan 5, 2018

CarbonDataQA commented Jan 5, 2018

ravipesala commented Jan 5, 2018

mohammadshahidkhan commented Jun 23, 2017 •

edited