[CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data #2488

ndwangsen · 2018-07-11T10:30:12Z

[CARBONDATA-2724]Unsupported create datamap on table with V1 or V2 format data

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
NA
Any backward compatibility impacted?
NA
Document update required?
NA
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
test pass in environment
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

CarbonDataQA · 2018-07-11T13:54:38Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7025/

CarbonDataQA · 2018-07-11T17:17:59Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5805/

ravipesala · 2018-07-11T20:51:24Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5776/

ndwangsen · 2018-07-12T11:32:18Z

retest this please

ndwangsen · 2018-07-12T12:17:02Z

retest sdv please

CarbonDataQA · 2018-07-12T13:39:45Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7057/

CarbonDataQA · 2018-07-12T14:34:21Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5834/

ravipesala · 2018-07-12T19:38:14Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5803/

ndwangsen · 2018-07-13T01:12:00Z

retest sdv please

CarbonDataQA · 2018-07-13T06:20:02Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7084/

CarbonDataQA · 2018-07-13T07:35:43Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5861/

ndwangsen · 2018-07-13T11:45:35Z

retest sdv please

ravipesala · 2018-07-14T00:45:41Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5833/

jackylk · 2018-07-15T14:06:00Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

+            if (file == null) {
+              return false;
+            }
+            return file.getName().endsWith("carbonindex");


I think it is better to get the version from data file instead of index file which is an optional file

@ravipesala Is there any utility func to get the version from CarbonTable object already?

It should not be under carbontable object. It is file version and it is related to each load, not table. @manishgupta88 I think your PR already handles the validation of datamap loading on old formats right?

@ravipesala ...If dataMap is created on legacy store then in my PR I am throwing exception at the time block/blocklet pruning...but the dataMap creation blocking is not taken care in my PR...

so how can we get the version info for a table?

I have modified the comments to get version from the data file

ravipesala · 2018-07-16T05:19:37Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

+   */
+  public static ColumnarFormatVersion getFormatVersion(CarbonTable carbonTable) throws IOException
+  {
+    String tablePath = carbonTable.getTablePath();


This check would be wrong, carbon files not always be under table path directly.

xuchuanyin · 2018-07-16T06:44:57Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

+   * carbon Table
+   */
+  public static ColumnarFormatVersion getFormatVersion(CarbonTable carbonTable) throws IOException
+  {


Why this format can pass the checkstyle?

chenliang613 · 2018-07-16T06:55:47Z

retest this please

CarbonDataQA · 2018-07-16T10:31:37Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7209/

CarbonDataQA · 2018-07-16T11:01:49Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5984/

CarbonDataQA · 2018-07-16T13:57:53Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7215/

jackylk · 2018-07-16T14:43:26Z

retest sdv please

CarbonDataQA · 2018-07-16T16:07:16Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5991/

ravipesala · 2018-07-17T00:16:09Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5877/

ndwangsen · 2018-07-17T02:34:23Z

retest this please

CarbonDataQA · 2018-07-17T03:02:10Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6002/

xuchuanyin · 2018-07-17T03:26:27Z

retest this please

format data

CarbonDataQA · 2018-07-17T04:34:19Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7231/

CarbonDataQA · 2018-07-17T04:40:56Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6004/

ndwangsen · 2018-07-17T06:01:31Z

retest sdv please

CarbonDataQA · 2018-07-17T06:35:00Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7234/

ndwangsen · 2018-07-17T06:48:33Z

retest this please

CarbonDataQA · 2018-07-17T07:33:08Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6007/

ndwangsen · 2018-07-17T09:02:25Z

retest this please

ravipesala · 2018-07-17T11:11:16Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5885/

CarbonDataQA · 2018-07-17T11:19:50Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7244/

xuchuanyin · 2018-07-17T14:26:08Z

LGTM

CarbonDataQA · 2018-07-17T14:45:33Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6018/

CarbonDataQA · 2018-07-17T18:01:58Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7256/

CarbonDataQA · 2018-07-18T00:07:36Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6032/

ravipesala · 2018-07-18T04:29:06Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

+    // if the carbontable is support flat folder
+    boolean supportFlatFolder = carbonTable.isSupportFlatFolder();
+    if (supportFlatFolder) {
+      storePath = carbonTable.getTablePath();


It does not work for partition tables.

… or V2 format data block creating datamap on carbon table with V1 or V2 format Currently the version info is read from carbon data file This closes #2488

jackylk reviewed Jul 15, 2018

View reviewed changes

ravipesala reviewed Jul 16, 2018

View reviewed changes

xuchuanyin reviewed Jul 16, 2018

View reviewed changes

ndwangsen force-pushed the dm_block_v1_v2 branch from c16331b to f2dfe69 Compare July 16, 2018 09:08

ndwangsen force-pushed the dm_block_v1_v2 branch from f2dfe69 to 95fed69 Compare July 17, 2018 02:24

[CARBONDATA-2724]Unsupported create datamap on table with V1 or V2

afed660

format data

ndwangsen force-pushed the dm_block_v1_v2 branch from 95fed69 to afed660 Compare July 17, 2018 04:08

asfgit closed this in a162897 Jul 17, 2018

ravipesala reviewed Jul 18, 2018

View reviewed changes

[CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data #2488

[CARBONDATA-2724][DataMap]Unsupported create datamap on table with V1 or V2 format data #2488

Conversation

ndwangsen commented Jul 11, 2018 • edited Loading

CarbonDataQA commented Jul 11, 2018

CarbonDataQA commented Jul 11, 2018

ravipesala commented Jul 11, 2018

ndwangsen commented Jul 12, 2018

ndwangsen commented Jul 12, 2018

CarbonDataQA commented Jul 12, 2018

CarbonDataQA commented Jul 12, 2018

ravipesala commented Jul 12, 2018

ndwangsen commented Jul 13, 2018

CarbonDataQA commented Jul 13, 2018

CarbonDataQA commented Jul 13, 2018

ndwangsen commented Jul 13, 2018

ravipesala commented Jul 14, 2018

Choose a reason for hiding this comment

jackylk Jul 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenliang613 commented Jul 16, 2018

CarbonDataQA commented Jul 16, 2018

CarbonDataQA commented Jul 16, 2018

CarbonDataQA commented Jul 16, 2018

jackylk commented Jul 16, 2018

CarbonDataQA commented Jul 16, 2018

ravipesala commented Jul 17, 2018

ndwangsen commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

xuchuanyin commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

ndwangsen commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

ndwangsen commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

ndwangsen commented Jul 17, 2018

ravipesala commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

xuchuanyin commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

CarbonDataQA commented Jul 17, 2018

CarbonDataQA commented Jul 18, 2018

Choose a reason for hiding this comment

ndwangsen commented Jul 11, 2018 •

edited

Loading

jackylk Jul 15, 2018 •

edited

Loading