[WIP] Datamap implementation for Blocklet. #1075
Conversation
|
Refer to this link for build results (access rights to CI server needed): |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2644/ |
|
|
||
| /** | ||
| * It is called to initialize and load the required table datamap metadata. | ||
| */ | ||
| void init(AbsoluteTableIdentifier identifier, String dataMapName); | ||
| public abstract void init(AbsoluteTableIdentifier identifier, String dataMapName); | ||
|
|
||
| /** | ||
| * Gives the writer to write the metadata information of this datamap at table level. |
There was a problem hiding this comment.
Please use "Get the datamap writer" to replace "Gives the writer"
| /** | ||
| * Store the data to unsafe | ||
| */ | ||
| public class UnsafeMemoryDMStore { |
There was a problem hiding this comment.
Can you add comment to explain: store what kind of data ?
|
Build Failed with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/121/ |
|
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2695/ |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.23 MB...][ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn -rf :carbondata-presto[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examples-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomchannel stoppedSetting status of 34f2acd to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/620/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6 |
ravipesala
force-pushed
the
blocklet-datamap
branch
from
34f2acd
to
36471a2
Compare
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2696/ |
|
Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/122/ |
|
Refer to this link for build results (access rights to CI server needed): |
ravipesala
force-pushed
the
blocklet-datamap
branch
from
36471a2
to
b9983cc
Compare
|
Build Success with Spark 1.6, Please check CI http://144.76.159.231:8080/job/ApacheCarbonPRBuilder/141/ |
|
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2716/ |
|
Refer to this link for build results (access rights to CI server needed): |
Bypassed btree implementations and used datamap to get the pruned blocklets. Only driver side datamap is maintained and all required information will be send to executor.
No datamap/btree will be maintained in executor.
Implementation classes are
Here only used unsafe storage to store data.