[CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table #3022

manishnalla1994 · 2018-12-24T12:23:52Z

Problem1: Compaction Failing for Complex datatypes with Dictionary Include as KeyGenenrator was not being set in model for Dictionary Include Complex Columns and dictionary include complex columns were not handled for finding cardinality.

Solution: Handled both these issues by setting KeyGenerator and storing cardinality of Complex dictionary include columns.

Problem2: Compaction was failing for restructured table containing dictionary include complex columns.

Solution: Handled complex columns for this case by inserting correct indices of the columns.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-12-24T12:34:03Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1929/

CarbonDataQA · 2018-12-24T12:44:05Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1930/

CarbonDataQA · 2018-12-24T13:53:50Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10183/

CarbonDataQA · 2018-12-24T14:10:12Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2140/

CarbonDataQA · 2018-12-27T01:33:38Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1952/

manishgupta88 · 2018-12-27T05:34:32Z

...rg/apache/carbondata/integration/spark/testsuite/complexType/TestCompactionComplexType.scala

@@ -1068,4 +1068,40 @@ class TestCompactionComplexType extends QueryTest with BeforeAndAfterAll {
    sql("Drop table if exists adaptive")
  }

+  test("Test major compaction for struct of array type") {
+    sql("DROP TABLE IF EXISTS carbon")


Include dropping of table in afterAll also

Give the test case description as below
Test major compaction with dictionary include for struct of array type

manishgupta88 · 2018-12-27T05:43:16Z

processing/src/main/java/org/apache/carbondata/processing/store/CarbonFactDataHandlerModel.java

+        new int[segmentProperties.getDimColumnsCardinality().length + segmentProperties
+            .getComplexDimColumnCardinality().length];
+    for (int i = 0; i < segmentProperties.getDimColumnsCardinality().length; i++) {
+      dimAndComplexColumnCardinality[i] = segmentProperties.getDimColumnsCardinality()[i];


Check for a resturcture drop column case when few loads are done and then dictionary column is dropped and compaction is triggered. In that case segmentProperties will contain cardinality of dropped column also check finally what is the schema and cardinality written during compaction

The restructure case is not handled for complex types compaction. I have raised the JIRA issue and I will handle it. Please find the JIRA link : 'https://issues.apache.org/jira/browse/CARBONDATA-3203' .

manishgupta88 · 2018-12-27T05:52:12Z

processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java

@@ -699,4 +701,33 @@ public static boolean isRawDataRequired(CarbonDataLoadConfiguration configuratio
    return iterators;
  }

+  public static int[] calcDimensionLengths(int numberOfSortColumns, int[] complexCardinality) {
+    if (!(numberOfSortColumns > 0)) {


Rewrite the condition if (!(numberOfSortColumns > 0)) as if (numberOfSortColumns == 0)

The functionality of this method is not clear. Add a comment to explain the logic explanation and use of this method

manishgupta88 · 2018-12-27T05:52:35Z

processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java

+    for (int eachDimLen : complexCardinality) {
+      if (eachDimLen != 0) dimsLenList.add(eachDimLen);
+    }
+    int[] dimLens = new int[dimsLenList.size()];


Conversion from arrayList to array can be done directly

manishgupta88 · 2018-12-27T05:54:27Z

processing/src/main/java/org/apache/carbondata/processing/util/CarbonDataProcessorUtil.java

+    return dimLens;
+  }
+
+  public static KeyGenerator[] createKeyGeneratorForComplexDimension(int numberOfSortColumns,


Add a method comment to explain the logic and method usage

CarbonDataQA · 2018-12-27T07:52:53Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10205/

CarbonDataQA · 2018-12-27T11:03:54Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2227/

CarbonDataQA · 2018-12-27T19:12:46Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2257/

CarbonDataQA · 2018-12-27T23:56:54Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2051/

CarbonDataQA · 2018-12-28T03:01:52Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10303/

CarbonDataQA · 2018-12-28T08:42:00Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2062/

CarbonDataQA · 2018-12-28T08:47:38Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2266/

CarbonDataQA · 2018-12-28T09:42:00Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10315/

manishgupta88 · 2018-12-28T11:33:04Z

LGTM

… with Dictionary Include and also supported Compaction for restructured table Problem1: Compaction Failing for Complex datatypes with Dictionary Include as KeyGenenrator was not being set in model for Dictionary Include Complex Columns and dictionary include complex columns were not handled for finding cardinality. Solution: Handled both these issues by setting KeyGenerator and storing cardinality of Complex dictionary include columns. Problem2: Compaction was failing for restructured table containing dictionary include complex columns. Solution: Handled complex columns for this case by inserting correct indices of the columns. This closes #3022

… with Dictionary Include and also supported Compaction for restructured table Problem1: Compaction Failing for Complex datatypes with Dictionary Include as KeyGenenrator was not being set in model for Dictionary Include Complex Columns and dictionary include complex columns were not handled for finding cardinality. Solution: Handled both these issues by setting KeyGenerator and storing cardinality of Complex dictionary include columns. Problem2: Compaction was failing for restructured table containing dictionary include complex columns. Solution: Handled complex columns for this case by inserting correct indices of the columns. This closes apache#3022

manishnalla1994 force-pushed the ComplexCompactionIssue branch 2 times, most recently from 5e975de to 0045616 Compare December 24, 2018 12:32

manishgupta88 suggested changes Dec 27, 2018

View reviewed changes

manishnalla1994 force-pushed the ComplexCompactionIssue branch from 0045616 to 2011ecb Compare December 27, 2018 12:55

manishnalla1994 changed the title ~~[CARBONDATA-3196] Fixed Compaction for Complex types with Dictionary Include~~ [CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also support Compaction for restructured table Dec 27, 2018

Fixed Compaction for Complex types with Dictionary Include

b715677

manishnalla1994 force-pushed the ComplexCompactionIssue branch from 2011ecb to b715677 Compare December 28, 2018 07:15

asfgit closed this in 7c4e79f Dec 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table #3022

[CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table #3022

manishnalla1994 commented Dec 24, 2018 •

edited

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 27, 2018

manishgupta88 Dec 27, 2018

manishgupta88 Dec 27, 2018

manishnalla1994 Dec 27, 2018

manishgupta88 Dec 27, 2018

manishgupta88 Dec 27, 2018

manishgupta88 Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

manishgupta88 commented Dec 28, 2018

[CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table #3022

[CARBONDATA-3196] [CARBONDATA-3203]Fixed Compaction for Complex types with Dictionary Include and also supported Compaction for restructured table #3022

Conversation

manishnalla1994 commented Dec 24, 2018 • edited

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 27, 2018

manishgupta88 Dec 27, 2018

Choose a reason for hiding this comment

manishgupta88 Dec 27, 2018

Choose a reason for hiding this comment

manishnalla1994 Dec 27, 2018

Choose a reason for hiding this comment

manishgupta88 Dec 27, 2018

Choose a reason for hiding this comment

manishgupta88 Dec 27, 2018

Choose a reason for hiding this comment

manishgupta88 Dec 27, 2018

Choose a reason for hiding this comment

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

manishgupta88 commented Dec 28, 2018

manishnalla1994 commented Dec 24, 2018 •

edited