[CarbonData-1250] change default partition id & Add TaskId group in partitionInfo #1119

lionelcao · 2017-06-29T16:07:50Z

This PR change the default partition id from Max to 0 and add taskId management related code to support later add(split)/drop(merge) partition operations.

change default partition id from Max to 0
add taskIdInPartitionOrder in partitionInfo to manage the taskId to partition map
remove some useless file
extract some common used function to PartitionUtils
change CarbonInputFormat to use correct taskId
modify test case according to the taskId change.
support range info in ascending order validation
fix some typo

asfgit · 2017-06-29T16:07:52Z

Can one of the admins verify this patch?

asfgit · 2017-06-29T16:07:52Z

Can one of the admins verify this patch?

CarbonDataQA · 2017-06-29T16:09:27Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/228/

CarbonDataQA · 2017-06-29T16:23:33Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2809/

chenliang613 · 2017-06-29T23:05:03Z

add to whitelist

CarbonDataQA · 2017-06-29T23:06:30Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2818/

CarbonDataQA · 2017-06-29T23:10:24Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/238/

CarbonDataQA · 2017-07-01T14:52:28Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2838/

CarbonDataQA · 2017-07-01T14:55:27Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/258/

CarbonDataQA · 2017-07-01T15:09:48Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/259/

CarbonDataQA · 2017-07-01T15:11:15Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2839/

CarbonDataQA · 2017-07-01T16:19:15Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2841/

CarbonDataQA · 2017-07-01T16:19:49Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/261/

CarbonDataQA · 2017-07-02T10:00:24Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2846/

CarbonDataQA · 2017-07-02T10:00:28Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/265/

CarbonDataQA · 2017-07-02T10:08:24Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2847/

CarbonDataQA · 2017-07-02T10:10:23Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/266/

CarbonDataQA · 2017-07-02T10:15:00Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2848/

CarbonDataQA · 2017-07-02T10:16:00Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/267/

CarbonDataQA · 2017-07-02T10:25:09Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2849/

CarbonDataQA · 2017-07-02T10:26:08Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/268/

CarbonDataQA · 2017-07-02T11:15:11Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2850/

CarbonDataQA · 2017-07-02T11:16:12Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/269/

CarbonDataQA · 2017-07-02T11:21:10Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/270/

CarbonDataQA · 2017-07-02T11:21:13Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2851/

CarbonDataQA · 2017-07-02T13:11:11Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/271/

CarbonDataQA · 2017-07-02T13:20:14Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2853/

CarbonDataQA · 2017-07-02T13:34:50Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/273/

CarbonDataQA · 2017-07-02T13:35:12Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2854/

CarbonDataQA · 2017-07-02T15:24:49Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/274/

CarbonDataQA · 2017-07-02T15:25:06Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/275/

CarbonDataQA · 2017-07-02T15:25:14Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2855/

CarbonDataQA · 2017-07-02T15:25:32Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2856/

CarbonDataQA · 2017-07-02T15:25:50Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2857/

CarbonDataQA · 2017-07-02T15:38:27Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2858/

CarbonDataQA · 2017-07-02T15:40:26Z

Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/276/

QiangCai · 2017-07-03T08:52:19Z

can you rebase and combine all commits to a commit

QiangCai · 2017-07-03T09:30:52Z

...ain/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java

@@ -219,6 +219,10 @@
    externalPartitionInfo.setList_info(wrapperPartitionInfo.getListInfo());
    externalPartitionInfo.setRange_info(wrapperPartitionInfo.getRangeInfo());
    externalPartitionInfo.setNum_partitions(wrapperPartitionInfo.getNumPartitions());
+    externalPartitionInfo.setNumOfPartitions(wrapperPartitionInfo.getNumberOfPartitions());


what is different between Num_partitions with NumOfPartitions?

QiangCai · 2017-07-03T09:32:23Z

...ain/java/org/apache/carbondata/core/metadata/converter/ThriftWrapperSchemaConverterImpl.java

@@ -219,6 +219,10 @@
    externalPartitionInfo.setList_info(wrapperPartitionInfo.getListInfo());
    externalPartitionInfo.setRange_info(wrapperPartitionInfo.getRangeInfo());
    externalPartitionInfo.setNum_partitions(wrapperPartitionInfo.getNumPartitions());
+    externalPartitionInfo.setNumOfPartitions(wrapperPartitionInfo.getNumberOfPartitions());
+    externalPartitionInfo.setMax_partition(wrapperPartitionInfo.getMAX_PARTITION());
+    externalPartitionInfo.setTaskId_In_PartitionOrder(wrapperPartitionInfo


why need to save taskId?

QiangCai · 2017-07-03T09:51:19Z

core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java

      } else {
        // LessThan(<), LessThanEqualTo(<=)
-        partitionMap.set(0, partitioner.numPartitions());
+        partitionMap.set(1, partitioner.numPartitions());


no need to change it.
It means all partitions were matched.

QiangCai · 2017-07-03T09:55:09Z

core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java

      }
    } else {
      // filter value is not in default partition
      if (result == 0) {
        // if result is 0, the filter value is a bound value of range partition.
        if (isGreaterThan) {
          // GreaterThan(>), GreaterThanEqualTo(>=)
-          partitionMap.set(partitionIndex + 1, partitioner.numPartitions());
+          partitionMap.set(partitionIndex + 1, partitioner.numPartitions() - 1);


partitionMap.set(partitionIndex + 2, partitioner.numPartitions());

QiangCai · 2017-07-03T09:56:18Z

core/src/main/java/org/apache/carbondata/core/scan/filter/partition/PartitionFilterUtil.java

          }
        }
      } else {
        // the filter value is not a bound value of range partition
        if (isGreaterThan) {
          // GreaterThan(>), GreaterThanEqualTo(>=)
-          partitionMap.set(partitionIndex, partitioner.numPartitions());
+          partitionMap.set(partitionIndex + 1, partitioner.numPartitions() - 1);


partitionMap.set(partitionIndex + 1, partitioner.numPartitions() );

lionelcao added 2 commits June 30, 2017 00:26

[CARBONDATA-1250] change default partition id from Max to 0

6a5f905

extract common function

ff477c5

lionelcao added 2 commits June 30, 2017 23:05

fix conflict

65db050

remove partition statistics for now

4195c3c

lionelcao force-pushed the carbon_910_6 branch from b5c3616 to 4195c3c Compare July 1, 2017 14:51

fix style error

53ef4f2

empty commit

e3793a7

lionelcao changed the title ~~[CarbonData-1205] change default partition id from Max to 0~~ [CarbonData-1205] change default partition id & Add TaskId group in partitionInfo Jul 2, 2017

add taskid group in partitionInfo

1ab132b

remove unrelated code

4c4200b

fix style error

b89269f

fix style error

69271e6

fix bug: getSplits error for non-partition table

0b40312

change import order

354b09e

remove unrelated code and test CI

be031af

change import order

7826ad2

lionelcao added 3 commits July 2, 2017 23:18

change query filter code according to default partition id changing

91c7800

thrift converter

f758892

remove useless comment

2ebeea4

remove unrelated code

344b394

lionelcao changed the title ~~[CarbonData-1205] change default partition id & Add TaskId group in partitionInfo~~ [CarbonData-1250] change default partition id & Add TaskId group in partitionInfo Jul 3, 2017

lionelcao closed this Jul 3, 2017

QiangCai reviewed Jul 3, 2017

View reviewed changes

[CarbonData-1250] change default partition id & Add TaskId group in partitionInfo #1119

[CarbonData-1250] change default partition id & Add TaskId group in partitionInfo #1119

Conversation

lionelcao commented Jun 29, 2017 • edited

asfgit commented Jun 29, 2017

asfgit commented Jun 29, 2017

CarbonDataQA commented Jun 29, 2017

CarbonDataQA commented Jun 29, 2017

chenliang613 commented Jun 29, 2017

CarbonDataQA commented Jun 29, 2017

CarbonDataQA commented Jun 29, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 1, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

CarbonDataQA commented Jul 2, 2017

QiangCai commented Jul 3, 2017

QiangCai Jul 3, 2017

Choose a reason for hiding this comment

QiangCai Jul 3, 2017

Choose a reason for hiding this comment

QiangCai Jul 3, 2017

Choose a reason for hiding this comment

QiangCai Jul 3, 2017

Choose a reason for hiding this comment

QiangCai Jul 3, 2017

Choose a reason for hiding this comment

lionelcao commented Jun 29, 2017 •

edited