[CARBONDATA-1971][Blocklet Prunning] Measure Null value blocklet prunning optimization #1751

sounakr · 2018-01-03T03:10:33Z

Recognise Null values in blocklets so that it can help in filter prunning.
Exclude filter isScanRequired is modified in order to help driver for blocklet prunning.

Any interfaces changed? Yes
Any backward compatibility impacted? No
Document update required? No
Testing done Yes
Please provide details on
- Whether new unit test cases have been added or why no new tests are required? Yes
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. NA

CarbonDataQA · 2018-01-03T04:04:29Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2496/

CarbonDataQA · 2018-01-03T04:59:15Z

Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1271/

ravipesala · 2018-01-03T05:50:55Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2658/

sounakr · 2018-01-03T06:36:21Z

Retest this please.

jackylk · 2018-01-03T06:56:20Z

core/src/main/java/org/apache/carbondata/core/datastore/chunk/AbstractRawColumnChunk.java

@@ -29,6 +29,8 @@

  private byte[][] maxValues;

+  private byte[] nullValues;


please add comment

jackylk · 2018-01-03T06:56:24Z

core/src/main/java/org/apache/carbondata/core/datastore/DataRefNode.java

@@ -71,6 +71,8 @@
   */
  byte[][] getColumnsMinValue();

+  byte[] getColumnsNullValue();


please add comment

sounakr · 2018-01-03T07:56:34Z

Retest this please.

CarbonDataQA · 2018-01-03T08:56:59Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2503/

CarbonDataQA · 2018-01-03T10:44:25Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1279/

CarbonDataQA · 2018-01-03T11:35:52Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2507/

CarbonDataQA · 2018-01-03T13:05:07Z

Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1282/

ravipesala · 2018-01-03T16:03:41Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2671/

ravipesala · 2018-01-03T16:40:11Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2672/

CarbonDataQA · 2018-01-04T18:39:03Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1322/

CarbonDataQA · 2018-01-04T21:09:33Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2560/

ravipesala · 2018-01-04T23:22:02Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2723/

ravipesala · 2018-01-05T08:41:02Z

format/src/main/thrift/carbondata.thrift

@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
 struct BlockletMinMaxIndex{
    1: required list<binary> min_values; //Min value of all columns of one blocklet Bit-Packed
    2: required list<binary> max_values; //Max value of all columns of one blocklet Bit-Packed
+    3: optional list<byte> null_value;   // Null Bool turn on if null value present.


Better use binary and try to use BitSet for null values

we can use PresenceMeta to store null value

Done. Stored as a Bitset

ravipesala · 2018-01-05T08:41:45Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala

@@ -39,8 +39,8 @@ object ExampleUtils {
    val warehouse = s"$rootPath/examples/spark2/target/warehouse"

    CarbonProperties.getInstance()
-      .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
-      .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
+      .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd HH:mm:ss")


Don't change the example class

ravipesala · 2018-01-05T08:42:29Z

examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala

-         | SELECT *
-         | FROM carbon_table
-         | WHERE stringfield = 'spark' AND decimalField > 40
+        | SELECT *


Don't change this example.

ravipesala · 2018-01-05T08:45:41Z

core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java

@@ -25,6 +25,8 @@

  Object getMax();

+  byte getNull();


Better use boolean and change method to isNull

We can do it. But in later phase of execution the Boolean will again get converted to Byte[]. So it doesn't make that much difference.

ravipesala · 2018-01-05T08:58:53Z

.../src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java

@@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[] measures) {
    return dimensionMaxValue;
  }

+  public byte[] getDimentionNullValue() {


typo getDimensionNullValue

ravipesala · 2018-01-05T09:00:57Z

core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java

      List<String> partitions = new ArrayList<>();
-      DataMapRow row = unsafeRow.getRow(2);
+      DataMapRow row = unsafeRow.getRow(3);


Please add constant to partition index

ravipesala · 2018-01-05T09:01:25Z

...ain/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java

@@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List<TableBlockInfo> blockInfos, int index,
    return null;
  }

+  @Override public byte[] getColumnsNullValue() {


Change byte[] to BitSet

ravipesala · 2018-01-05T09:05:29Z

core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java

@@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax)
   * @param blockMinValue
   * @return BitSet
   */
-  BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue);
+  BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue);


Better wrap these parameters to new BlockletStats object

ravipesala · 2018-01-05T09:06:10Z

.../main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java

@@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal
   *
   * @return
   */
-  Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue);
+  Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue,


Here also use BlockletStats object

ravipesala · 2018-01-05T09:06:14Z

.../main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java

@@ -34,7 +34,7 @@
   * @return
   */
  BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue,
-      String uniqueBlockPath);
+      byte[] nullValue, String uniqueBlockPath);


Here also use BlockletStats object

CarbonDataQA · 2018-01-19T13:55:56Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2992/

CarbonDataQA · 2018-01-19T13:55:56Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1762/

ravipesala · 2018-01-22T12:51:41Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3031/

CarbonDataQA · 2018-01-22T12:52:24Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3036/

CarbonDataQA · 2018-01-22T12:55:15Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1805/

CarbonDataQA · 2018-02-02T07:14:03Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3432/

CarbonDataQA · 2018-03-14T16:01:52Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3050/

CarbonDataQA · 2018-03-14T17:21:08Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4294/

CarbonDataQA · 2018-04-19T19:42:39Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5183/

CarbonDataQA · 2018-04-19T21:55:07Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3973/

ravipesala · 2018-04-22T17:49:59Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4466/

CarbonDataQA · 2018-07-14T02:53:10Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7125/

CarbonDataQA · 2018-07-14T08:14:47Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5901/

CarbonDataQA · 2018-09-05T16:01:10Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/53/

CarbonDataQA · 2018-09-26T19:03:20Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8802/

CarbonDataQA · 2018-09-26T21:57:36Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/735/

CarbonDataQA · 2018-09-26T22:31:57Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/557/

CarbonDataQA · 2018-11-28T11:25:48Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1781/

CarbonDataQA · 2018-11-29T16:06:27Z

Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9846/

CarbonDataQA · 2018-12-27T04:22:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2005/

CarbonDataQA · 2018-12-27T08:14:03Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2182/

CarbonDataQA · 2018-12-27T21:54:06Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10257/

CarbonDataQA · 2019-05-28T16:32:10Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/3564/

CarbonDataQA · 2019-07-24T09:09:34Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/80/

CarbonDataQA · 2019-07-24T09:39:45Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/79/

CarbonDataQA1 · 2020-02-03T07:07:05Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/79/

CarbonDataQA2 · 2021-10-07T08:24:06Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6020/

sounakr changed the title ~~[WIP]Measure null~~ [CARBONDATA-1971][Blocklet Prunning] Measure Null value blocklet prunning optimization Jan 3, 2018

jackylk reviewed Jan 3, 2018

View reviewed changes

sounakr force-pushed the measure_null branch 2 times, most recently from f1be641 to 9efbc5b Compare January 3, 2018 07:56

sounakr force-pushed the measure_null branch from 9efbc5b to e4172db Compare January 4, 2018 12:27

ravipesala reviewed Jan 5, 2018

View reviewed changes

sounakr force-pushed the measure_null branch from 26e9a82 to 5d5ca3a Compare January 19, 2018 12:50

Test Case

409aace

asfgit force-pushed the master branch from 23a9e7c to e07df44 Compare September 26, 2018 07:48

		@@ -29,6 +29,8 @@

		private byte[][] maxValues;

		private byte[] nullValues;

[CARBONDATA-1971][Blocklet Prunning] Measure Null value blocklet prunning optimization #1751

Are you sure you want to change the base?

[CARBONDATA-1971][Blocklet Prunning] Measure Null value blocklet prunning optimization #1751

Conversation

sounakr commented Jan 3, 2018 • edited

CarbonDataQA commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

ravipesala commented Jan 3, 2018

sounakr commented Jan 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sounakr commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

CarbonDataQA commented Jan 3, 2018

ravipesala commented Jan 3, 2018

ravipesala commented Jan 3, 2018

CarbonDataQA commented Jan 4, 2018

CarbonDataQA commented Jan 4, 2018

ravipesala commented Jan 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Jan 19, 2018

CarbonDataQA commented Jan 19, 2018

ravipesala commented Jan 22, 2018

CarbonDataQA commented Jan 22, 2018

CarbonDataQA commented Jan 22, 2018

CarbonDataQA commented Feb 2, 2018

CarbonDataQA commented Mar 14, 2018

CarbonDataQA commented Mar 14, 2018

CarbonDataQA commented Apr 19, 2018

CarbonDataQA commented Apr 19, 2018

ravipesala commented Apr 22, 2018

CarbonDataQA commented Jul 14, 2018

CarbonDataQA commented Jul 14, 2018

CarbonDataQA commented Sep 5, 2018

CarbonDataQA commented Sep 26, 2018

CarbonDataQA commented Sep 26, 2018

CarbonDataQA commented Sep 26, 2018

CarbonDataQA commented Nov 28, 2018

CarbonDataQA commented Nov 29, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented May 28, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA1 commented Feb 3, 2020

CarbonDataQA2 commented Oct 7, 2021

sounakr commented Jan 3, 2018 •

edited