[CARBONDATA-1018] Add unsafe ColumnPage implementation #1000

jackylk · 2017-06-06T11:07:45Z

This PR is based on #987

In this PR, an UnsafeColumnPage is added, it can reduce memory requirement for data loading.
Before loading, user can add following property to enable this feature

    CarbonProperties.getInstance()
      .addProperty(CarbonCommonConstants.ENABLE_LOADING_UNSAFE_COLUMN_PAGE, "true")

asfgit · 2017-06-06T11:08:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/105/

CarbonDataQA · 2017-06-06T11:12:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2232/

CarbonDataQA · 2017-06-06T13:14:25Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2237/

asfgit · 2017-06-06T13:23:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/110/

asfgit · 2017-06-08T16:45:02Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/177/

Failed Tests: 11

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark: 2

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 9

CarbonDataQA · 2017-06-08T16:46:41Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2304/

CarbonDataQA · 2017-06-08T17:28:40Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2307/

asfgit · 2017-06-08T18:08:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/180/

CarbonDataQA · 2017-06-09T00:40:35Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2308/

asfgit · 2017-06-09T00:44:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/181/

asfgit · 2017-06-09T12:55:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/199/

CarbonDataQA · 2017-06-09T13:06:35Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2322/

jackylk · 2017-06-09T13:23:27Z

apache:retest this please

CarbonDataQA · 2017-06-09T13:34:52Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2324/

asfgit · 2017-06-09T13:41:00Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/201/

asfgit · 2017-06-10T00:26:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/211/

Failed Tests: 1

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1

org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query

CarbonDataQA · 2017-06-10T00:35:30Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2334/

CarbonDataQA · 2017-06-13T17:44:07Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2452/

asfgit · 2017-06-13T17:52:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/333/

chenerlu · 2017-06-14T01:54:53Z

core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java

@@ -98,56 +117,54 @@ public static ColumnPage newPage(DataType dataType, int pageSize) {
      default:
        throw new RuntimeException("Unsupported data dataType: " + dataType);
    }
-    instance.stats = new ColumnPageStatsVO(dataType);
-    instance.nullBitSet = new BitSet(pageSize);
    return instance;
  }

  protected static ColumnPage newBytePage(byte[] byteData) {
    ColumnPage columnPage = new ColumnPage(BYTE, byteData.length);


BytePage does not support unsafeColumnPage ?

ok, I will fix

chenerlu · 2017-06-14T01:55:12Z

core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java

@@ -98,56 +117,54 @@ public static ColumnPage newPage(DataType dataType, int pageSize) {
      default:
        throw new RuntimeException("Unsupported data dataType: " + dataType);
    }
-    instance.stats = new ColumnPageStatsVO(dataType);
-    instance.nullBitSet = new BitSet(pageSize);
    return instance;
  }

  protected static ColumnPage newBytePage(byte[] byteData) {
    ColumnPage columnPage = new ColumnPage(BYTE, byteData.length);


Byte doe not support unsafeColumnPage ?

If don't support, we'd better to add some notes to explain.

ok, it supports, I will fix

chenerlu · 2017-06-14T02:04:50Z

core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java

-    ColumnPage columnPage = new ColumnPage(BYTE_ARRAY, stringData.length);
-    columnPage.byteArrayData = stringData;
+  private static ColumnPage newVarLengthPage(byte[][] byteArray) {
+    ColumnPage columnPage = new ColumnPage(BYTE_ARRAY, byteArray.length);


ByteArray does not support unsafeColumnPage ?

It supports, I will update

chenerlu · 2017-06-14T03:40:56Z

core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java

@@ -508,7 +567,7 @@ public static ColumnPage decompress(Compressor compressor, DataType dataType,
  }

  // input byte[] is LV encoded, this function can expand it into byte[][]
-  private static byte[][] deflatten(byte[] input) {
+  protected static byte[][] deflatten(byte[] input) {


Will it be better to move deflatten to ByteUtil which is same with flatten ?

Agree with erlu.

ravipesala · 2017-06-15T03:46:32Z

core/src/main/java/org/apache/carbondata/core/datastore/page/UnsafeVarLengthColumnPage.java

+  private void ensureMemory(int requestSize) throws MemoryException {
+    if (totalLength + requestSize > capacity) {
+      memoryBlock = UnsafeMemoryManager.reallocateMemoryWithRetry(
+          memoryBlock, (long)((totalLength + requestSize) * FACTOR));


capacity should be updated with (totalLength + requestSize) * FACTOR

ravipesala · 2017-06-15T04:09:32Z

core/src/main/java/org/apache/carbondata/core/datastore/page/UnsafeVarLengthColumnPage.java

+    CarbonUnsafe.unsafe.copyMemory(bytes, CarbonUnsafe.BYTE_ARRAY_OFFSET + offset,
+        baseAddress, baseOffset + rowOffset[rowId], length);
+  }
+


Even method public void putBytes(int rowId, byte[] bytes) also need to be override here.

It is implemented in VarLengthColumnPageBase and it will call putBytesAtRow abstract method

yes, i see that. I mean we should do in unsafe

CarbonDataQA · 2017-06-15T06:57:20Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2491/

CarbonDataQA · 2017-06-15T07:18:37Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2494/

asfgit · 2017-06-15T09:56:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/387/

asfgit · 2017-06-15T10:08:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/390/

Failed Tests: 8

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 8

CarbonDataQA · 2017-06-15T10:31:48Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2509/

CarbonDataQA · 2017-06-15T13:47:22Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2521/

asfgit · 2017-06-15T14:04:53Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/406/

CarbonDataQA · 2017-06-15T14:59:37Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2526/

asfgit · 2017-06-15T14:59:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/414/

asfgit · 2017-06-15T15:15:55Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/415/

CarbonDataQA · 2017-06-15T15:23:29Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2528/

jackylk · 2017-06-16T09:44:46Z

retest this please

CarbonDataQA · 2017-06-16T10:04:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2546/

asfgit · 2017-06-16T10:16:37Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/438/

CarbonDataQA · 2017-06-16T15:10:48Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2551/

asfgit · 2017-06-16T15:16:42Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/443/

Failed Tests: 2

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 2

org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option and compaction
org.apache.carbondata.spark.testsuite.dataload.TestBatchSortDataLoad.test batch sort load by passing option in one load and with out option in other load and then do compaction

jackylk · 2017-06-17T01:51:44Z

retest this please

CarbonDataQA · 2017-06-17T02:02:31Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2558/

asfgit · 2017-06-17T02:11:20Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/450/

Failed Tests: 1

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1

org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query

CarbonDataQA · 2017-06-19T04:16:52Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2574/

asfgit · 2017-06-19T04:16:56Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/468/

ravipesala · 2017-06-19T05:49:58Z

LGTM

…1000

jackylk force-pushed the unsafepage branch from be95455 to 419291f Compare June 6, 2017 13:12

jackylk force-pushed the unsafepage branch from 419291f to f2890a9 Compare June 8, 2017 16:33

jackylk force-pushed the unsafepage branch from f2890a9 to fe8fb85 Compare June 8, 2017 17:06

jackylk force-pushed the unsafepage branch from fe8fb85 to c1c8d37 Compare June 9, 2017 00:29

jackylk force-pushed the unsafepage branch from c1c8d37 to 25a62bd Compare June 9, 2017 12:54

jackylk force-pushed the unsafepage branch from 25a62bd to 3e2cd44 Compare June 10, 2017 00:07

jackylk mentioned this pull request Jun 10, 2017

[CARBONDATA-1015] Extract interface in data load write step #971

Merged

jackylk force-pushed the unsafepage branch from 3e2cd44 to 925686e Compare June 13, 2017 17:34

chenerlu reviewed Jun 14, 2017

View reviewed changes

ravipesala reviewed Jun 15, 2017

View reviewed changes

jackylk force-pushed the unsafepage branch from 923d000 to 81ce9e1 Compare June 15, 2017 09:45

jackylk force-pushed the unsafepage branch from 81ce9e1 to 7d54d72 Compare June 15, 2017 13:36

add unsafe column page

7359601

jackylk force-pushed the unsafepage branch from eebd2db to 7359601 Compare June 19, 2017 04:01

asfgit merged commit 7359601 into apache:master Jun 19, 2017

asfgit pushed a commit that referenced this pull request Jun 19, 2017

[CARBONDATA-1018] Add unsafe ColumnPage implementation. This closes #…

d5e3000

…1000

[CARBONDATA-1018] Add unsafe ColumnPage implementation #1000

[CARBONDATA-1018] Add unsafe ColumnPage implementation #1000

Conversation

jackylk commented Jun 6, 2017 • edited

asfgit commented Jun 6, 2017

CarbonDataQA commented Jun 6, 2017

CarbonDataQA commented Jun 6, 2017

asfgit commented Jun 6, 2017

asfgit commented Jun 8, 2017

Failed Tests: 11

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark: 2

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 9

CarbonDataQA commented Jun 8, 2017

CarbonDataQA commented Jun 8, 2017

asfgit commented Jun 8, 2017

CarbonDataQA commented Jun 9, 2017

asfgit commented Jun 9, 2017

asfgit commented Jun 9, 2017

CarbonDataQA commented Jun 9, 2017

jackylk commented Jun 9, 2017

CarbonDataQA commented Jun 9, 2017

asfgit commented Jun 9, 2017

asfgit commented Jun 10, 2017

Failed Tests: 1

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1

CarbonDataQA commented Jun 10, 2017

CarbonDataQA commented Jun 13, 2017

asfgit commented Jun 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackylk Jun 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Jun 15, 2017

CarbonDataQA commented Jun 15, 2017

asfgit commented Jun 15, 2017

asfgit commented Jun 15, 2017

Failed Tests: 8

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 8

CarbonDataQA commented Jun 15, 2017

CarbonDataQA commented Jun 15, 2017

asfgit commented Jun 15, 2017

CarbonDataQA commented Jun 15, 2017

asfgit commented Jun 15, 2017

asfgit commented Jun 15, 2017

CarbonDataQA commented Jun 15, 2017

jackylk commented Jun 16, 2017

CarbonDataQA commented Jun 16, 2017

asfgit commented Jun 16, 2017

CarbonDataQA commented Jun 16, 2017

asfgit commented Jun 16, 2017

Failed Tests: 2

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 2

jackylk commented Jun 17, 2017

CarbonDataQA commented Jun 17, 2017

asfgit commented Jun 17, 2017

Failed Tests: 1

carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test: 1

CarbonDataQA commented Jun 19, 2017

asfgit commented Jun 19, 2017

ravipesala commented Jun 19, 2017

jackylk commented Jun 6, 2017 •

edited

jackylk Jun 15, 2017 •

edited