[WIP] Implement range interval partition #1105
Conversation
|
Can one of the admins verify this patch? |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2758/ |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 64.18 KB...][ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :carbondata-spark-common[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examples-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomchannel stoppedSetting status of 90bbd1a to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/687/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6 |
|
Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/179/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2759/ |
|
Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/180/ |
|
Refer to this link for build results (access rights to CI server needed): |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2789/ |
|
Refer to this link for build results (access rights to CI server needed): |
|
Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/210/ |
| package org.apache.carbondata.core.metadata.schema.partition; | ||
|
|
||
|
|
||
| public class RangeIntervalPartition extends AbstractPartition { |
There was a problem hiding this comment.
AbstractPartition has been removed. Now only store rangeInfo/ListInfo in partitionInfo.
| partitionMap.set(0); | ||
| if (isGreaterThan) { | ||
| // GreaterThan(>), GreaterThanEqualTo(>=) | ||
| partitionMap.set(partitionId, partitioner.numPartitions()); |
There was a problem hiding this comment.
this should be partition index
| @@ -188,7 +189,7 @@ object CommonUtil { | |||
| rangeInfo.get.split(",").map(_.trim).foreach( | |||
| isValid &= validateTypeConvert(partitionerFields(0), _)) | |||
| } | |||
| case "RANGE_INTERVAL" => isValid = false | |||
| case "RANGE_INTERVAL" => isValid = rangeIntervalInfo.isDefined | |||
There was a problem hiding this comment.
Is there any validate function for range interval info?
| break; | ||
| } | ||
| } | ||
| return partitionId; |
There was a problem hiding this comment.
this should be partition index
| break; | ||
| } | ||
| } | ||
| return partitionId; |
There was a problem hiding this comment.
this should be partition index
| break; | ||
| } | ||
| } | ||
| return partitionId; |
There was a problem hiding this comment.
this should be partition index
| break; | ||
| } | ||
| } | ||
| return partitionId; |
There was a problem hiding this comment.
this should be partition index
| break; | ||
| } | ||
| } | ||
| return partitionId; |
There was a problem hiding this comment.
this should be partition index
| } | ||
| if (partitionIndex == numPartitions) { | ||
| // filter value is not in given partition, should get partitionId dynamically | ||
| int partitionId = partitioner.getPartition(filterValue); |
There was a problem hiding this comment.
this should be partition index
|
Can one of the admins verify this patch? |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/44/ |
|
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1221/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/590/ |
This PR is try to implement range interval partition and now work on process.