[CARBONDATA-2388][SDK]Avro Record Complex Type Implementation #2209

sounakr · 2018-04-23T04:53:12Z

Avro Complex DataType Support.

AVRO Complex type. Supported Datatype - ARRAYS and RECORDS.
Carbon Supported DataType - ARRAYS and STRUCT
SDK support to handle complex datatype.
Carbon Complex Type Support
- Support for NonDictionary Fields.
- Existing Complex type bug fixes.
Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-04-23T05:29:37Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4138/

CarbonDataQA · 2018-04-23T06:46:46Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5319/

jackylk · 2018-04-23T12:28:16Z

Please modify the PR title, it is for SDK, change to [SDK]

CarbonDataQA · 2018-04-28T16:05:13Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5510/

CarbonDataQA · 2018-04-28T16:06:35Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4345/

ravipesala · 2018-04-28T19:10:05Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4611/

CarbonDataQA · 2018-04-29T17:14:49Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4354/

CarbonDataQA · 2018-04-29T17:17:47Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5518/

ravipesala · 2018-04-29T19:30:55Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4620/

ravipesala · 2018-04-29T19:41:00Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4621/

CarbonDataQA · 2018-04-29T19:57:15Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4355/

CarbonDataQA · 2018-04-29T20:00:57Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5519/

ajantha-bhat · 2018-04-30T02:11:37Z

...cala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala

+//
+//    checkAnswer(sql("select * from sdkOutputTable"), Seq(Row("robot0", 0, 0.0),
+//      Row("robot1", 1, 0.5),
+//      Row("robot2", 2, 1.0)))


please revert back this test case and add new one

ajantha-bhat · 2018-04-30T02:29:49Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/SchemaReader.java

    // This routine is going to infer schema from the carbondata file footer
    // Convert the ColumnSchema -> TableSchema -> TableInfo.
    // Return the TableInfo.
+    if (tableInfoFromCache != null) {


If tableInfoFromCache is found, No need to call inferschema itself, please handle this outside infer schema

ravipesala · 2018-04-30T02:32:50Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4622/

CarbonDataQA · 2018-04-30T03:18:39Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4357/

CarbonDataQA · 2018-04-30T03:20:14Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5521/

ravipesala · 2018-04-30T03:21:21Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4623/

ravipesala · 2018-04-30T04:40:16Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4624/

CarbonDataQA · 2018-04-30T04:41:37Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4358/

CarbonDataQA · 2018-04-30T04:44:46Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5522/

kumarvishal09 · 2018-04-30T06:14:00Z

core/src/main/java/org/apache/carbondata/core/scan/complextypes/PrimitiveQueryType.java

+  }
+
+
+  public PrimitiveQueryType(String name, String parentname, int blockIndex,


One more extra constructor is not required. If dictionary is not null then isDictionary=true

kumarvishal09 · 2018-04-30T06:18:53Z

processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java

-      KeyGenerator[] generator) throws IOException, KeyGenException {
+  @Override public void parseAndBitPack(ByteBuffer byteArrayInput,
+      DataOutputStream dataOutputStream, KeyGenerator[] generator)
+      throws IOException, KeyGenException {
    int dataLength = byteArrayInput.getInt();



remove extra space

ravipesala · 2018-04-30T06:18:58Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/column/ColumnSchema.java

@@ -43,6 +43,9 @@
   */
  private static final long serialVersionUID = 7676766554874863763L;

+  public void columnSchema() {


What is the use of this method

Removed. No Use.

kumarvishal09 · 2018-04-30T06:20:45Z

processing/src/main/java/org/apache/carbondata/processing/datatypes/ArrayDataType.java

-  @Override
-  public void parseAndBitPack(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
-      KeyGenerator[] generator) throws IOException, KeyGenException {
+  @Override public void parseAndBitPack(ByteBuffer byteArrayInput,


Please remove this method from interface as below new method will be used

ravipesala · 2018-04-30T06:22:09Z

...main/java/org/apache/carbondata/core/scan/collector/impl/DictionaryBasedResultCollector.java

+            .getDataBasedOnDataTypeFromNoDictionary(
+                ByteBuffer.wrap(complexTypeKeyArray[complexTypeColumnIndex++]));
+      } else
+      {


Move the braces up

ravipesala · 2018-04-30T06:54:45Z

...sing/src/main/java/org/apache/carbondata/processing/loading/CarbonDataLoadConfiguration.java

@@ -348,6 +356,33 @@ public void setCardinalityFinder(DictionaryCardinalityFinder cardinalityFinder)
    return complexKeyGenerators;
  }

+  public Boolean[][] createComplexDictionaryFieldIdentification() {


If not required, please remove this method

ravipesala · 2018-04-30T06:57:29Z

processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java

@@ -222,8 +222,9 @@ private void addComplexColumn(int index, int rowId, byte[] complexColumns) {
      ByteBuffer byteArrayInput = ByteBuffer.wrap(complexColumns);
      ByteArrayOutputStream byteArrayOutput = new ByteArrayOutputStream();
      DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutput);
-      complexDataType.parseAndBitPack(byteArrayInput, dataOutputStream,
-          model.getComplexDimensionKeyGenerator());
+      int startOffset = 0;


What is the use of startOffset when not used?

ravipesala · 2018-04-30T06:58:42Z

processing/src/main/java/org/apache/carbondata/processing/datatypes/GenericDataType.java

+
+  int parseComplexValue(ByteBuffer byteArrayInput, DataOutputStream dataOutputStream,
+      KeyGenerator[] generator, Boolean[][] complexDictionaryIndentification, int startOffset)
+      throws IOException, KeyGenException;


Remove complexDictionaryIndentification and startOffset if not used

complexDictionaryIndentification is removed but StartOffset is used.

ravipesala · 2018-04-30T07:01:40Z

@sounakr Please add testcases for complextype also not only in AVRO.

CarbonDataQA · 2018-04-30T15:20:46Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5542/

ravipesala · 2018-04-30T15:26:13Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4637/

CarbonDataQA · 2018-04-30T16:11:02Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4379/

ravipesala · 2018-04-30T16:18:02Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4638/

CarbonDataQA · 2018-05-01T11:18:43Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5558/

CarbonDataQA · 2018-05-01T11:20:47Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4395/

ravipesala · 2018-05-01T12:41:51Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4654/

CarbonDataQA · 2018-05-01T13:59:40Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5560/

ravipesala · 2018-05-01T14:19:25Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4656/

CarbonDataQA · 2018-05-01T14:37:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5563/

CarbonDataQA · 2018-05-01T14:39:04Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4401/

CarbonDataQA · 2018-05-01T14:43:12Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4398/

ravipesala · 2018-05-01T14:44:32Z

@sounakr Please fix the build

a) Complex Type suport for AVRO Record and Array datatype. b) Complex Type No Dictionary Implementation.

sounakr · 2018-05-01T15:25:08Z

Retest this please

CarbonDataQA · 2018-05-01T16:18:44Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5565/

ravipesala · 2018-05-01T16:40:33Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4659/

CarbonDataQA · 2018-05-01T17:39:47Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4403/

ravipesala · 2018-05-01T17:55:24Z

LGTM

ravipesala · 2018-05-02T05:30:27Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4661/

Avro Complex DataType Support. AVRO Complex type. Supported Datatype - ARRAYS and RECORDS. Carbon Supported DataType - ARRAYS and STRUCT SDK support to handle complex datatype. Carbon Complex Type Support - Support for NonDictionary Fields. - Existing Complex type bug fixes. This closes apache#2209

sounakr force-pushed the avro-datatype branch from 464ade6 to 0cad5c1 Compare April 28, 2018 15:24

sounakr force-pushed the avro-datatype branch from 1381e66 to 92acbcb Compare April 30, 2018 01:40

ajantha-bhat reviewed Apr 30, 2018

View reviewed changes

sounakr force-pushed the avro-datatype branch from 92acbcb to 469b03b Compare April 30, 2018 02:23

sounakr changed the title ~~[WIP][Non Transactional Table]Avro Record Complex Type Implementation~~ [CARBONDATA-2388][SDK]Avro Record Complex Type Implementation Apr 30, 2018

ajantha-bhat reviewed Apr 30, 2018

View reviewed changes

kumarvishal09 reviewed Apr 30, 2018

View reviewed changes

ravipesala reviewed Apr 30, 2018

View reviewed changes

kumarvishal09 reviewed Apr 30, 2018

View reviewed changes

ravipesala reviewed Apr 30, 2018

View reviewed changes

sounakr force-pushed the avro-datatype branch from 478305f to eba8155 Compare April 30, 2018 13:43

sounakr force-pushed the avro-datatype branch from 61a6692 to 1f6238b Compare May 1, 2018 12:38

sounakr force-pushed the avro-datatype branch from 1f6238b to 1c2a8c8 Compare May 1, 2018 13:08

[CARBONDATA-2388][SDK] Avro Record Complex Type Implementation.

36b49ef

a) Complex Type suport for AVRO Record and Array datatype. b) Complex Type No Dictionary Implementation.

sounakr force-pushed the avro-datatype branch from 1c2a8c8 to 36b49ef Compare May 1, 2018 15:24

asfgit closed this in 3202cf5 May 1, 2018

		}


		public PrimitiveQueryType(String name, String parentname, int blockIndex,

[CARBONDATA-2388][SDK]Avro Record Complex Type Implementation #2209

[CARBONDATA-2388][SDK]Avro Record Complex Type Implementation #2209

Conversation

sounakr commented Apr 23, 2018 • edited

CarbonDataQA commented Apr 23, 2018

CarbonDataQA commented Apr 23, 2018

jackylk commented Apr 23, 2018

CarbonDataQA commented Apr 28, 2018

CarbonDataQA commented Apr 28, 2018

ravipesala commented Apr 28, 2018

CarbonDataQA commented Apr 29, 2018

CarbonDataQA commented Apr 29, 2018

ravipesala commented Apr 29, 2018

ravipesala commented Apr 29, 2018

CarbonDataQA commented Apr 29, 2018

CarbonDataQA commented Apr 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravipesala commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

ravipesala commented Apr 30, 2018

ravipesala commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

Choose a reason for hiding this comment

sounakr Apr 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravipesala commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

ravipesala commented Apr 30, 2018

CarbonDataQA commented Apr 30, 2018

ravipesala commented Apr 30, 2018

CarbonDataQA commented May 1, 2018

CarbonDataQA commented May 1, 2018

ravipesala commented May 1, 2018

CarbonDataQA commented May 1, 2018

ravipesala commented May 1, 2018

CarbonDataQA commented May 1, 2018

CarbonDataQA commented May 1, 2018

CarbonDataQA commented May 1, 2018

ravipesala commented May 1, 2018

sounakr commented May 1, 2018

CarbonDataQA commented May 1, 2018

ravipesala commented May 1, 2018

CarbonDataQA commented May 1, 2018

ravipesala commented May 1, 2018

ravipesala commented May 2, 2018

sounakr commented Apr 23, 2018 •

edited

sounakr Apr 30, 2018 •

edited