[CARBONDATA-1537] Fixed version compatabilty issues from V1 to latest carbon version #1398

ravipesala · 2017-09-30T12:07:59Z

No description provided.

CarbonDataQA · 2017-09-30T12:22:33Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/218/

CarbonDataQA · 2017-09-30T12:27:23Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/342/

ravipesala · 2017-09-30T12:44:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/970/

CarbonDataQA · 2017-10-01T10:47:14Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/343/

CarbonDataQA · 2017-10-01T10:47:33Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/219/

CarbonDataQA · 2017-10-01T11:05:48Z

Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/220/

ravipesala · 2017-10-01T11:08:15Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/971/

CarbonDataQA · 2017-10-01T11:08:59Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/344/

ravipesala · 2017-10-01T11:42:54Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/972/

CarbonDataQA · 2017-10-01T12:23:55Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/345/

CarbonDataQA · 2017-10-01T12:25:51Z

Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/221/

CarbonDataQA · 2017-10-01T12:42:17Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/346/

CarbonDataQA · 2017-10-01T12:43:40Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/222/

ravipesala · 2017-10-01T12:59:53Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/973/

ravipesala · 2017-10-01T13:33:54Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/974/

CarbonDataQA · 2017-10-01T15:07:43Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/223/

CarbonDataQA · 2017-10-01T15:12:24Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/347/

CarbonDataQA · 2017-10-01T16:42:37Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/224/

CarbonDataQA · 2017-10-01T16:47:08Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/348/

CarbonDataQA · 2017-10-01T17:42:17Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/349/

CarbonDataQA · 2017-10-01T17:42:39Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/225/

CarbonDataQA · 2017-10-01T18:22:21Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/350/

CarbonDataQA · 2017-10-02T06:17:18Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/354/

CarbonDataQA · 2017-10-02T06:17:29Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/230/

CarbonDataQA · 2017-10-02T08:12:30Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/355/

CarbonDataQA · 2017-10-02T08:13:16Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/231/

CarbonDataQA · 2017-10-02T10:12:44Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/232/

CarbonDataQA · 2017-10-02T10:17:24Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/356/

CarbonDataQA · 2017-10-02T10:47:13Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/357/

CarbonDataQA · 2017-10-02T10:47:34Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/233/

ravipesala · 2017-10-02T11:10:51Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/985/

jackylk · 2017-10-10T01:20:28Z

core/src/main/java/org/apache/carbondata/core/metadata/blocklet/BlockletInfo.java

@@ -214,6 +218,42 @@ public void setNumberOfPages(int numberOfPages) {
    for (int i = 0; i < mSize; i++) {
      output.writeInt(measureChunksLength.get(i));
    }
+    // Serialize datachunks as well for older versions like V1 and V2


Can you wrap this logic into a function and mentioning it is for V1 and V2 serialization only, I think it will be more readable

jackylk · 2017-10-10T01:20:59Z

core/src/main/java/org/apache/carbondata/core/metadata/blocklet/BlockletInfo.java

@@ -238,6 +278,20 @@ public void setNumberOfPages(int numberOfPages) {
    for (int i = 0; i < measureChunkOffsetsSize; i++) {
      measureChunksLength.add(input.readInt());
    }
-
+    // Deserialize datachunks as well for older versions like V1 and V2


Can you wrap this logic into a function and mentioning it is for V1 and V2 deserialization only, I think it will be more readable

jackylk · 2017-10-10T01:21:43Z

core/src/main/java/org/apache/carbondata/core/metadata/datatype/DataType.java

  // internal use only, for variable length data type
  BYTE_ARRAY(13, "BYTE_ARRAY", -1),
+


remove empty line

jackylk · 2017-10-10T01:22:08Z

core/src/main/java/org/apache/carbondata/core/metadata/datatype/DataType.java

  // internal use only, for value compression from integer/long to 3 bytes value
-  SHORT_INT(14, "SHORT_INT", 3);
+  SHORT_INT(14, "SHORT_INT", 3),
+  // Only for internal use for backward compatability


mentioning it is for V1 and V2 format only

jackylk · 2017-10-10T01:26:39Z

core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java

@@ -2070,6 +1994,20 @@ public static void dropDatabaseDirectory(String dbName, String storePath)
    }
  }

+  public static DataType getDataType(char type) {


Suggest to move all conversion to DataType enum, including this one and ColumnPageEncoderMeta.convertType. And also move CarbonCommonConstants.BIG_INT_MEASURE and related constants to DataType enum also.

jackylk · 2017-10-10T01:28:42Z

core/src/main/java/org/apache/carbondata/core/util/DataFileFooterConverter.java

@@ -123,6 +123,25 @@ private BlockletInfo getBlockletInfo(
  }

  @Override public List<ColumnSchema> getSchema(TableBlockInfo tableBlockInfo) throws IOException {
-    return null;
+    FileHolder fileReader = null;


Is this change related to this PR? Can you add comment to describe the intention of this function in its abstract interface?

yes, it is part of this PR. We need to get columnschema

jackylk · 2017-10-10T01:30:28Z

...st/scala/org/apache/carbondata/cluster/sdv/generated/CarbonV1toV3CompatabilityTestCase.scala

+import org.apache.carbondata.core.util.CarbonProperties
+
+/**
+ * V1 to V3 compatability test. This test has to be at last


Will you add V2 to V3 compatibility test in future PR?

yes, i will tests for V2 to V3 in another PR, Because I need to verify the compatibility of V2 to V3 first

jackylk · 2017-10-10T01:32:07Z

integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala

@@ -43,7 +43,7 @@ class CarbonSession(@transient val sc: SparkContext,
  }

  @transient
-  override private[sql] lazy val sessionState: SessionState = new CarbonSessionState(this)
+  override lazy val sessionState: SessionState = new CarbonSessionState(this)


Is this change required? Somewhere outside sql package will read this variable?

Yes, it is required for test case I added.

CarbonDataQA · 2017-10-10T19:02:35Z

Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/294/

CarbonDataQA · 2017-10-10T19:06:52Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/418/

ravipesala · 2017-10-10T19:25:16Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1048/

jackylk · 2017-10-11T13:24:10Z

LGTM

… carbon version This closes apache#1398

Fixed version compatabilty issues from V1 to latest carbon version

4a0d82f

ravipesala force-pushed the upgrade branch from 2683969 to 56e78c6 Compare October 1, 2017 11:02

ravipesala force-pushed the upgrade branch from 56e78c6 to d0300b4 Compare October 1, 2017 12:21

ravipesala force-pushed the upgrade branch from d0300b4 to 2a243dd Compare October 1, 2017 12:26

ravipesala force-pushed the upgrade branch from 2a243dd to 358b013 Compare October 1, 2017 14:52

ravipesala force-pushed the upgrade branch from 358b013 to 97ebdaf Compare October 1, 2017 16:27

ravipesala force-pushed the upgrade branch from 97ebdaf to 5bf313d Compare October 1, 2017 17:26

ravipesala force-pushed the upgrade branch from 5bf313d to a97cbdc Compare October 1, 2017 18:05

ravipesala force-pushed the upgrade branch from 130292e to 2465f93 Compare October 2, 2017 06:00

ravipesala force-pushed the upgrade branch from 2465f93 to 888df96 Compare October 2, 2017 07:54

ravipesala force-pushed the upgrade branch from 888df96 to edbd666 Compare October 2, 2017 09:57

Added comptability test for V1 to V3 format

ecbea70

ravipesala force-pushed the upgrade branch from edbd666 to ecbea70 Compare October 2, 2017 10:31

ravipesala changed the title ~~[WIP] Fixed version compatabilty issues from V1 to latest carbon version~~ [CARBONDATA-1537] Fixed version compatabilty issues from V1 to latest carbon version Oct 5, 2017

jackylk reviewed Oct 10, 2017

View reviewed changes

Fixed comments

556ae3c

asfgit closed this in 133b303 Oct 11, 2017

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-1537] Fixed version compatabilty issues from V1 to latest…

ac3f452

… carbon version This closes apache#1398

		// internal use only, for variable length data type
		BYTE_ARRAY(13, "BYTE_ARRAY", -1),

[CARBONDATA-1537] Fixed version compatabilty issues from V1 to latest carbon version #1398

[CARBONDATA-1537] Fixed version compatabilty issues from V1 to latest carbon version #1398

Conversation

ravipesala commented Sep 30, 2017

CarbonDataQA commented Sep 30, 2017

CarbonDataQA commented Sep 30, 2017

ravipesala commented Sep 30, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

ravipesala commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

ravipesala commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

ravipesala commented Oct 1, 2017

ravipesala commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 1, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

CarbonDataQA commented Oct 2, 2017

ravipesala commented Oct 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Oct 10, 2017

CarbonDataQA commented Oct 10, 2017

ravipesala commented Oct 10, 2017

jackylk commented Oct 11, 2017