[CARBONDATA-1304] [IUD Bug] Iud with single pass #1167

mohammadshahidkhan · 2017-07-12T15:53:53Z

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Problem:
- The Update on carbon table is failing with single pass. In case of single pass the tupleId is not being
arranged in the end.
solution:
- In case of single pass the tupleId should be retrieved using SegIdUDF function and should be
arranged in the end.

Any interfaces changed?
None
Any backward compatibility impacted?
None
Document update required?
None
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required? Yes
- How it is tested? Please attach test report.
Added the Functional Test case for update with persist false.
Added functional test case for update with single pass true
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
None

asfgit · 2017-07-12T15:53:56Z

Can one of the admins verify this patch?

asfgit · 2017-07-12T15:53:56Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-07-12T15:55:16Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3052/

CarbonDataQA · 2017-07-12T15:59:15Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3053/

CarbonDataQA · 2017-07-12T16:02:41Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/464/

CarbonDataQA · 2017-07-12T16:04:05Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/465/

mohammadshahidkhan · 2017-07-13T08:34:07Z

retest this please

CarbonDataQA · 2017-07-13T08:35:38Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/477/

CarbonDataQA · 2017-07-13T08:40:38Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/478/

CarbonDataQA · 2017-07-13T08:46:26Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3066/

CarbonDataQA · 2017-07-13T09:00:54Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3067/

CarbonDataQA · 2017-07-13T10:43:19Z

Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/487/

CarbonDataQA · 2017-07-13T10:49:20Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/3076/

CarbonDataQA · 2017-08-02T08:45:37Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-09-13T00:08:48Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/35/

ravipesala · 2017-10-23T09:13:03Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1213/

CarbonDataQA · 2017-10-23T14:53:52Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/582/

ravipesala · 2017-10-24T00:17:51Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1287/

CarbonDataQA · 2017-10-24T01:03:50Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/655/

mohammadshahidkhan · 2017-10-31T07:21:42Z

retest this please

CarbonDataQA · 2017-10-31T09:04:51Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/768/

CarbonDataQA · 2017-10-31T09:44:23Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/770/

CarbonDataQA · 2017-11-08T13:16:42Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/882/

ravipesala · 2017-11-08T13:39:24Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1494/

mohammadshahidkhan · 2017-11-28T11:28:15Z

retest this please

CarbonDataQA · 2017-11-28T12:58:38Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1537/

CarbonDataQA · 2017-11-29T17:15:44Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1579/

ravipesala · 2017-11-29T18:46:24Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1974/

CarbonDataQA · 2017-12-09T06:11:55Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1848/

CarbonDataQA · 2017-12-09T06:13:22Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/620/

CarbonDataQA · 2017-12-09T09:51:54Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1849/

CarbonDataQA · 2017-12-09T09:53:29Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/621/

mohammadshahidkhan · 2017-12-11T09:06:46Z

retest this please

CarbonDataQA · 2017-12-11T09:46:26Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/637/

CarbonDataQA · 2017-12-11T09:51:42Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1866/

ravipesala · 2017-12-11T16:21:34Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2216/

CarbonDataQA · 2017-12-14T15:01:31Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/742/

CarbonDataQA · 2017-12-14T16:21:10Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1974/

ravipesala · 2017-12-14T18:46:36Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2290/

manishgupta88 · 2017-12-15T06:55:16Z

.../scala/org/apache/carbondata/spark/testsuite/iud/TestUpdateCarbonTableWithPersistFalse.scala

+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestUpdateCarbonTableWithPersistFalse extends QueryTest with BeforeAndAfterAll {


Dont add a new test case. Add this test case in the existing update table test case class

@manishgupta88 Thanks for the review Fixed

manishgupta88 · 2017-12-15T06:55:42Z

...st/scala/org/apache/carbondata/spark/testsuite/iud/TestUpdateCarbonTableWithSinglePass.scala

+import org.apache.carbondata.core.constants.{CarbonCommonConstants, CarbonLoadOptionConstants}
+import org.apache.carbondata.core.util.CarbonProperties
+
+class TestUpdateCarbonTableWithSinglePass extends QueryTest with BeforeAndAfterAll {


Dont add a new test case. Add this test case in the existing update table test case class

@manishgupta88 Thanks for the review Fixed

manishgupta88 · 2017-12-15T06:59:58Z

...src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala

@@ -314,14 +315,19 @@ case class CarbonLoadDataCommand(
    } else {
      None
    }
+    var loadDataFrame = dataFrame


Modify the code to make the variable as Val
val loadDataFrame = if (updateModel.isDefined) {
Some(getDataFrameWithTupleID)
} else {
dataFrame
}

@manishgupta88 Thanks for the review Fixed

CarbonDataQA · 2017-12-15T11:04:51Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/774/

ravipesala · 2017-12-15T12:34:14Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2319/

CarbonDataQA · 2017-12-15T15:44:13Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2013/

manishgupta88 · 2017-12-16T06:28:09Z

LGTM

The Update on carbon table is failing with single pass. In case of single pass the tupleId is not being arranged in the end. In case of single pass the tupleId should be retrieved using SegIdUDF function and should be arranged in the end. This closes apache#1167

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 7c0c452 to 480b24b Compare July 12, 2017 15:56

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 480b24b to 37ebd26 Compare July 13, 2017 08:30

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 37ebd26 to c128a75 Compare July 13, 2017 10:22

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from c128a75 to 0b6367e Compare October 23, 2017 13:52

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 0b6367e to 213202a Compare October 31, 2017 06:39

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 213202a to cfa5571 Compare November 8, 2017 11:59

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from e2e7b6e to b1c5d48 Compare November 29, 2017 12:30

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from b1c5d48 to ca3097e Compare December 9, 2017 05:50

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from ca3097e to 1cc8e62 Compare December 9, 2017 09:30

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from 1cc8e62 to a2c7e93 Compare December 14, 2017 12:35

manishgupta88 suggested changes Dec 15, 2017

View reviewed changes

[CARBONDATA-1304] Support IUD with single pass

d8d408f

mohammadshahidkhan force-pushed the iud_withsingle_pass branch from a2c7e93 to d8d408f Compare December 15, 2017 09:11

mohammadshahidkhan changed the title ~~[CARBONDATA-1304] [IUD BuggFix] Iud with single pass~~ [CARBONDATA-1304] [IUD Bug] Iud with single pass Dec 15, 2017

asfgit closed this in 8bf72a6 Dec 16, 2017

[CARBONDATA-1304] [IUD Bug] Iud with single pass #1167

[CARBONDATA-1304] [IUD Bug] Iud with single pass #1167

Conversation

mohammadshahidkhan commented Jul 12, 2017 • edited

asfgit commented Jul 12, 2017

asfgit commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

CarbonDataQA commented Jul 12, 2017

mohammadshahidkhan commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Jul 13, 2017

CarbonDataQA commented Aug 2, 2017

CarbonDataQA commented Sep 13, 2017

ravipesala commented Oct 23, 2017

CarbonDataQA commented Oct 23, 2017

ravipesala commented Oct 24, 2017

CarbonDataQA commented Oct 24, 2017

mohammadshahidkhan commented Oct 31, 2017

CarbonDataQA commented Oct 31, 2017

CarbonDataQA commented Oct 31, 2017

CarbonDataQA commented Nov 8, 2017

ravipesala commented Nov 8, 2017

mohammadshahidkhan commented Nov 28, 2017

CarbonDataQA commented Nov 28, 2017

CarbonDataQA commented Nov 29, 2017

ravipesala commented Nov 29, 2017

CarbonDataQA commented Dec 9, 2017

CarbonDataQA commented Dec 9, 2017

CarbonDataQA commented Dec 9, 2017

CarbonDataQA commented Dec 9, 2017

mohammadshahidkhan commented Dec 11, 2017

CarbonDataQA commented Dec 11, 2017

CarbonDataQA commented Dec 11, 2017

ravipesala commented Dec 11, 2017

CarbonDataQA commented Dec 14, 2017

CarbonDataQA commented Dec 14, 2017

ravipesala commented Dec 14, 2017

manishgupta88 Dec 15, 2017

Choose a reason for hiding this comment

mohammadshahidkhan Dec 15, 2017

Choose a reason for hiding this comment

manishgupta88 Dec 15, 2017

Choose a reason for hiding this comment

mohammadshahidkhan Dec 15, 2017

Choose a reason for hiding this comment

manishgupta88 Dec 15, 2017

Choose a reason for hiding this comment

mohammadshahidkhan Dec 15, 2017

Choose a reason for hiding this comment

CarbonDataQA commented Dec 15, 2017

ravipesala commented Dec 15, 2017

CarbonDataQA commented Dec 15, 2017

manishgupta88 commented Dec 16, 2017

mohammadshahidkhan commented Jul 12, 2017 •

edited