[CARBONDATA-2430][SDK] Reshuffling of Columns given by user in SDK. #2261

sounakr · 2018-05-03T03:54:19Z

Reshuffling of Columns given by the user in SDK. Order should be Sort COlumns -> Dimension -> Complex --> Measure

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-05-03T04:48:24Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5598/

CarbonDataQA · 2018-05-03T04:52:57Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4437/

ravipesala · 2018-05-03T06:02:25Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4691/

sounakr · 2018-05-03T06:02:41Z

Retest this please

kumarvishal09 · 2018-05-03T07:23:20Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/TableSchemaBuilder.java

-        throw new IllegalArgumentException("column name already exists");
-      }
-    }
+//    for (ColumnSchema column : otherColumns) {


Duplicate column handling is required for non sort column, please handle it.

CarbonDataQA · 2018-05-03T07:35:12Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5602/

CarbonDataQA · 2018-05-03T07:36:32Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4441/

ravipesala · 2018-05-03T09:16:10Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4695/

CarbonDataQA · 2018-05-03T10:54:17Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4448/

CarbonDataQA · 2018-05-03T10:55:23Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5609/

xuchuanyin · 2018-05-03T14:04:50Z

It is reorder, not reshuffle. Better to modify the title of PR and commit

ravipesala · 2018-05-03T14:59:36Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4702/

ravipesala · 2018-05-03T15:47:48Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4703/

CarbonDataQA · 2018-05-03T17:32:44Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5623/

CarbonDataQA · 2018-05-03T17:46:22Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4463/

ravipesala · 2018-05-04T02:12:42Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4716/

sounakr · 2018-05-04T04:25:49Z

Retest this please

CarbonDataQA · 2018-05-04T05:18:05Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5630/

CarbonDataQA · 2018-05-04T05:18:42Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4470/

sounakr · 2018-05-04T06:34:45Z

Retest this please

sounakr · 2018-05-04T08:02:02Z

Retest this please

CarbonDataQA · 2018-05-04T08:41:21Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4477/

CarbonDataQA · 2018-05-04T09:38:03Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4480/

CarbonDataQA · 2018-05-04T10:12:08Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5640/

kumarvishal09 · 2018-05-04T10:14:22Z

store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java

+            tableSchemaBuilder.addColumn(new StructField(field.getFieldName(), complexType), false);
+          }
+        } else {
+


remove empty lines

ravipesala · 2018-05-04T10:17:14Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4719/

kunal642 · 2018-05-04T11:37:39Z

store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java

+          // unsupported types for ("array", "struct", "double", "float", "decimal")
+          if (field.getDataType() == DataTypes.DOUBLE || field.getDataType() == DataTypes.FLOAT
+              || DataTypes.isDecimal(field.getDataType()) || DataTypes
+              .isArrayType(field.getDataType()) || DataTypes.isStructType(field.getDataType())) {


use field.getDataType().isComplexType() to check for complex types

sounakr · 2018-05-04T12:10:27Z

Retest this please

CarbonDataQA · 2018-05-04T13:30:52Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5648/

CarbonDataQA · 2018-05-04T13:35:37Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4488/

ravipesala · 2018-05-04T14:24:08Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4724/

CarbonDataQA · 2018-05-04T16:57:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5655/

CarbonDataQA · 2018-05-04T17:53:07Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5657/

CarbonDataQA · 2018-05-04T18:19:34Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4497/

ravipesala · 2018-05-04T20:48:03Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4732/

ravipesala · 2018-05-04T22:27:11Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4734/

…onary a) Reshuffling of fields. Sorting of columns based on SORT -> DIMENSION -> COMPLEX -> MEASURE. b) Enable Complex Type NoDictionary Creation from Create Table DDL. c) ARRAY Support in AVRO. d) Minor Fixes related to AVRO SDK support.

sounakr · 2018-05-07T06:17:25Z

Retest this please

ravipesala · 2018-05-07T07:21:01Z

@sounakr Building schema should be common across spark-integration and sdk, there should not be two paths. I feel it is better to refactor to use common code.

CarbonDataQA · 2018-05-07T08:16:13Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5687/

CarbonDataQA · 2018-05-07T08:21:57Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4527/

sounakr · 2018-05-07T08:41:53Z

Retest this please.

sounakr · 2018-05-07T09:46:44Z

@ravipesala Refactoring of Building schema will be done as part of a separate PR as the changes are more and can impact the CarbonTable creation.

Jira CARBONDATA-2449 is created for tracking the above refactoring task.

CarbonDataQA · 2018-05-07T10:21:09Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5692/

CarbonDataQA · 2018-05-07T10:29:23Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4532/

ravipesala · 2018-05-07T11:01:43Z

LGTM

ravipesala · 2018-05-07T11:06:42Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4762/

ravipesala · 2018-05-07T13:00:42Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4764/

ravipesala · 2018-05-07T17:51:26Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4770/

Reshuffling of Columns given by the user in SDK. Order should be Sort COlumns -> Dimension -> Complex --> Measure This closes apache#2261

sounakr force-pushed the complex-type branch from e455f27 to 95dcebb Compare May 3, 2018 06:02

kumarvishal09 reviewed May 3, 2018

View reviewed changes

sounakr force-pushed the complex-type branch from 8047325 to 9ded564 Compare May 4, 2018 03:46

sounakr force-pushed the complex-type branch from 9ded564 to d366df2 Compare May 4, 2018 06:34

sounakr force-pushed the complex-type branch from d366df2 to 46fc3fe Compare May 4, 2018 08:01

kumarvishal09 reviewed May 4, 2018

View reviewed changes

kunal642 reviewed May 4, 2018

View reviewed changes

sounakr force-pushed the complex-type branch from 46fc3fe to 60f5efc Compare May 4, 2018 12:09

ajantha-bhat force-pushed the complex-type branch from 7f42621 to bbdab5e Compare May 7, 2018 06:13

sdk complex type issue fixes and test case additions

772fe46

sounakr force-pushed the complex-type branch from bbdab5e to 772fe46 Compare May 7, 2018 06:17

Review

004cd45

asfgit closed this in b1c85fa May 7, 2018

[CARBONDATA-2430][SDK] Reshuffling of Columns given by user in SDK. #2261

[CARBONDATA-2430][SDK] Reshuffling of Columns given by user in SDK. #2261

Conversation

sounakr commented May 3, 2018

CarbonDataQA commented May 3, 2018

CarbonDataQA commented May 3, 2018

ravipesala commented May 3, 2018

sounakr commented May 3, 2018

kumarvishal09 May 3, 2018

Choose a reason for hiding this comment

sounakr May 4, 2018

Choose a reason for hiding this comment

CarbonDataQA commented May 3, 2018

CarbonDataQA commented May 3, 2018

ravipesala commented May 3, 2018

CarbonDataQA commented May 3, 2018

CarbonDataQA commented May 3, 2018

xuchuanyin commented May 3, 2018

ravipesala commented May 3, 2018

ravipesala commented May 3, 2018

CarbonDataQA commented May 3, 2018

CarbonDataQA commented May 3, 2018

ravipesala commented May 4, 2018

sounakr commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

sounakr commented May 4, 2018

sounakr commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

kumarvishal09 May 4, 2018

Choose a reason for hiding this comment

sounakr May 4, 2018

Choose a reason for hiding this comment

ravipesala commented May 4, 2018

kunal642 May 4, 2018

Choose a reason for hiding this comment

sounakr May 4, 2018

Choose a reason for hiding this comment

sounakr commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

ravipesala commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

CarbonDataQA commented May 4, 2018

ravipesala commented May 4, 2018

ravipesala commented May 4, 2018

sounakr commented May 7, 2018

ravipesala commented May 7, 2018

CarbonDataQA commented May 7, 2018

CarbonDataQA commented May 7, 2018

sounakr commented May 7, 2018

sounakr commented May 7, 2018 • edited

CarbonDataQA commented May 7, 2018

CarbonDataQA commented May 7, 2018

ravipesala commented May 7, 2018

ravipesala commented May 7, 2018

ravipesala commented May 7, 2018

ravipesala commented May 7, 2018

sounakr commented May 7, 2018 •

edited