[CARBONDATA-2606]Fix Complex array Pushdown #2535

Indhumathi27 · 2018-07-21T10:49:35Z

What this PR for?
1.Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array.
2.Block Auto merge compaction for table containing complex datatype columns.
3.Fix Decimal Datatype scale and precision with two level struct type
4.Fix Dictionary Include for ComplexDataType
- If other complex columns other than first complex column is given in dictionary include, then its insertion fails.
5.Fix BadRecord and dateformat for Complex primitive type-DATE

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-07-21T11:54:27Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7373/

CarbonDataQA · 2018-07-21T12:11:59Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6134/

ravipesala · 2018-07-21T12:22:18Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5944/

kunal642 · 2018-07-23T06:50:17Z

...src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala

+      // Block compaction for table containing complex datatype
+      if (table.getTableInfo.getFactTable.getListOfColumns.asScala
+        .exists(m => m.getDataType.isComplexType)) {
+        LOGGER.info("Compaction is skipped as table contains complex columns")


change to warn

kunal642 · 2018-07-23T06:50:25Z

integration/spark2/src/main/scala/org/apache/carbondata/spark/rdd/CarbonDataRDDFactory.scala

+        // Block compaction for table containing complex datatype
+        if (carbonTable.getTableInfo.getFactTable.getListOfColumns.asScala
+          .exists(m => m.getDataType.isComplexType)) {
+          LOGGER.info("Compaction is skipped as table contains complex columns")


change to warn

kunal642 · 2018-07-23T06:52:17Z

...cala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala

+      "/Struct.csv' into table table1 options('delimiter'=','," +
+      "'quotechar'='\"','fileheader'='roll,person','complex_delimiter_level_1'='$'," +
+      "'complex_delimiter_level_2'='&')")
+    checkAnswer(sql("select count(*) from table1"),Seq(Row(40)))


check for segments whether compaction has happened or not

CarbonDataQA · 2018-07-23T08:15:00Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7389/

Indhumathi27 · 2018-07-23T09:35:28Z

Retest this please

CarbonDataQA · 2018-07-23T09:41:45Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6150/

CarbonDataQA · 2018-07-23T09:44:13Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7394/

CarbonDataQA · 2018-07-23T10:19:04Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6157/

Indhumathi27 · 2018-07-23T11:15:30Z

Retest this please

CarbonDataQA · 2018-07-23T13:20:47Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7401/

Indhumathi27 · 2018-07-23T13:21:30Z

Retest this please

CarbonDataQA · 2018-07-23T14:41:48Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6164/

ravipesala · 2018-07-23T14:46:03Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5963/

CarbonDataQA · 2018-07-23T15:36:22Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7407/

Indhumathi27 · 2018-07-23T15:46:40Z

Retest sdv please

CarbonDataQA · 2018-07-23T17:08:28Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6170/

CarbonDataQA · 2018-07-23T17:23:04Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7411/

kunal642 · 2018-07-24T06:58:21Z

...cala/org/apache/carbondata/integration/spark/testsuite/complexType/TestComplexDataType.scala

+      "'complex_delimiter_level_2'='&')")
+    checkExistence(sql("show segments for table table1"),false, "Compacted")
+    CarbonProperties.getInstance()
+      .addProperty(CarbonCommonConstants.ENABLE_AUTO_LOAD_MERGE, "false")


add this in afterAll also

CarbonDataQA · 2018-07-24T07:36:29Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7434/

CarbonDataQA · 2018-07-24T07:57:45Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6192/

Indhumathi27 · 2018-07-24T08:04:58Z

Retest this please

ravipesala · 2018-07-24T08:24:45Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5975/

brijoobopanna · 2018-07-24T08:38:57Z

retest this please

CarbonDataQA · 2018-07-24T08:52:35Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7441/

CarbonDataQA · 2018-07-24T09:59:10Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6198/

CarbonDataQA · 2018-07-24T19:18:22Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6214/

Indhumathi27 · 2018-07-24T19:21:00Z

Retest this please

CarbonDataQA · 2018-07-24T19:44:24Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7466/

CarbonDataQA · 2018-07-24T20:47:46Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7470/

CarbonDataQA · 2018-07-24T22:56:25Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6223/

CarbonDataQA · 2018-07-25T01:18:18Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6227/

Indhumathi27 · 2018-07-25T01:53:43Z

Retest this please

CarbonDataQA · 2018-07-25T03:14:26Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7474/

Indhumathi27 · 2018-07-25T03:26:13Z

Retest this please

CarbonDataQA · 2018-07-25T03:28:43Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6231/

CarbonDataQA · 2018-07-25T04:56:25Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7479/

kunal642 · 2018-07-25T05:19:05Z

...src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala

@@ -77,6 +77,7 @@ import org.apache.carbondata.spark.dictionary.provider.SecureDictionaryServicePr
 import org.apache.carbondata.spark.dictionary.server.SecureDictionaryServer
 import org.apache.carbondata.spark.load.{CsvRDDHelper, DataLoadProcessorStepOnSpark}
 import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory
+import org.apache.carbondata.spark.rdd.CarbonDataRDDFactory.LOGGER


no need for this import. CarbonLoadDataCommand already has a LOGGER

CarbonDataQA · 2018-07-25T05:35:09Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6236/

CarbonDataQA · 2018-07-25T06:34:16Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7483/

…action

CarbonDataQA · 2018-07-25T07:51:20Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6239/

CarbonDataQA · 2018-07-25T09:44:05Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7491/

CarbonDataQA · 2018-07-25T10:34:32Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6245/

kunal642 · 2018-07-25T12:17:22Z

LGTM

CarbonDataQA · 2018-07-25T13:34:05Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6252/

ravipesala · 2018-07-26T08:29:31Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5999/

…action 1.Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array. 2.Block Auto merge compaction for table containing complex datatype columns. 3.Fix Decimal Datatype scale and precision with two level struct type 4.Fix Dictionary Include for ComplexDataType - If other complex columns other than first complex column is given in dictionary include, then its insertion fails. 5.Fix BadRecord and dateformat for Complex primitive type-DATE This closes #2535

…action 1.Check for if Complex Column contains ArrayType at n levels and add parent to projection if contains array. 2.Block Auto merge compaction for table containing complex datatype columns. 3.Fix Decimal Datatype scale and precision with two level struct type 4.Fix Dictionary Include for ComplexDataType - If other complex columns other than first complex column is given in dictionary include, then its insertion fails. 5.Fix BadRecord and dateformat for Complex primitive type-DATE This closes apache#2535

Indhumathi27 force-pushed the arrayfix branch from e5b3d34 to 84f46b7 Compare July 23, 2018 06:46

kunal642 suggested changes Jul 23, 2018

View reviewed changes

Indhumathi27 force-pushed the arrayfix branch from 84f46b7 to 8ec3c99 Compare July 23, 2018 07:02

Indhumathi27 force-pushed the arrayfix branch from 8ec3c99 to 5626527 Compare July 24, 2018 06:28

kunal642 reviewed Jul 24, 2018

View reviewed changes

Indhumathi27 force-pushed the arrayfix branch from 5626527 to 392f5a3 Compare July 24, 2018 07:13

Indhumathi27 force-pushed the arrayfix branch from d48eef0 to f929cfd Compare July 24, 2018 15:10

Indhumathi27 force-pushed the arrayfix branch from f929cfd to 574ea5a Compare July 25, 2018 04:57

kunal642 reviewed Jul 25, 2018

View reviewed changes

Indhumathi27 force-pushed the arrayfix branch from 574ea5a to c94b494 Compare July 25, 2018 05:29

Indhumathi27 force-pushed the arrayfix branch from c94b494 to 0348eab Compare July 25, 2018 06:48

[CARBONDATA-2606]Fix Complex array Pushdown and block auto merge comp…

b407f0c

…action

Indhumathi27 force-pushed the arrayfix branch from 0348eab to b407f0c Compare July 25, 2018 07:40

kunal642 approved these changes Jul 25, 2018

View reviewed changes

asfgit closed this in 06d38ff Jul 25, 2018

[CARBONDATA-2606]Fix Complex array Pushdown #2535

[CARBONDATA-2606]Fix Complex array Pushdown #2535

Conversation

Indhumathi27 commented Jul 21, 2018 • edited

CarbonDataQA commented Jul 21, 2018

CarbonDataQA commented Jul 21, 2018

ravipesala commented Jul 21, 2018

kunal642 Jul 23, 2018

Choose a reason for hiding this comment

kunal642 Jul 23, 2018

Choose a reason for hiding this comment

kunal642 Jul 23, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jul 23, 2018

Indhumathi27 commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

Indhumathi27 commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

Indhumathi27 commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

ravipesala commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

Indhumathi27 commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

CarbonDataQA commented Jul 23, 2018

kunal642 Jul 24, 2018 • edited

Choose a reason for hiding this comment

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

Indhumathi27 commented Jul 24, 2018

ravipesala commented Jul 24, 2018

brijoobopanna commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

Indhumathi27 commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 24, 2018

CarbonDataQA commented Jul 25, 2018

Indhumathi27 commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

Indhumathi27 commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

kunal642 Jul 25, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

kunal642 commented Jul 25, 2018

CarbonDataQA commented Jul 25, 2018

ravipesala commented Jul 26, 2018

Indhumathi27 commented Jul 21, 2018 •

edited

kunal642 Jul 24, 2018 •

edited