[CARBONDATA-3189] Fix PreAggregate Datamap Issue #3010

Shubh18s · 2018-12-20T11:44:29Z

Problem -
Load and Select query was failing on table with preaggregate datamap.

Cause -
Previously if query on datamap was not enabled in thread, there was no check afterwards.

Solution -
After checking whether thread param for Direct Query On Datamap is enable. If not enable, we check in session params and then global.

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-12-20T11:53:56Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1877/

jackylk · 2018-12-20T12:15:29Z

Please create a JIRA issue and put it in the PR title

CarbonDataQA · 2018-12-20T12:56:25Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2087/

CarbonDataQA · 2018-12-20T13:03:53Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10132/

qiuchenjian · 2018-12-21T01:13:39Z

Can you show up the error message when load and query failing

Shubh18s · 2018-12-21T18:51:28Z

Here is the error message for loading

brijoobopanna · 2018-12-24T05:04:57Z

retest this please

brijoobopanna · 2018-12-24T05:05:03Z

add to whitelist

CarbonDataQA · 2018-12-24T05:22:08Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1918/

CarbonDataQA · 2018-12-24T06:13:08Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10171/

CarbonDataQA · 2018-12-24T06:13:24Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2128/

kumarvishal09 · 2018-12-24T06:55:42Z

@Shubh18s Please fix the test case failures

CarbonDataQA · 2018-12-27T02:01:47Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1960/

kunal642 · 2018-12-27T05:30:26Z

core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

@@ -1449,12 +1449,9 @@ private CarbonCommonConstants() {

  public static final String SUPPORT_DIRECT_QUERY_ON_DATAMAP_DEFAULTVALUE = "false";

-  @CarbonProperty
  public static final String VALIDATE_DIRECT_QUERY_ON_DATAMAP =
      "carbon.query.validate.direct.query.on.datamap";


I think we can remove this property and use carbon.query.directQueryOnDataMap.enabled.

Need to remove from configuration-paramters.md

Yes, I made changes to configuration-paramters.md accordingly and removed this property.

kunal642 · 2018-12-27T05:35:04Z

integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonLateDecodeRule.scala

      if (relation.carbonRelation.carbonTable.isChildDataMap) {
        isPreAggDataMapExists = true
      }
    }
    val validateQuery = CarbonProperties.getInstance
-      .getProperty(CarbonCommonConstants.VALIDATE_DIRECT_QUERY_ON_DATAMAP,
-        CarbonCommonConstants.VALIDATE_DIRECT_QUERY_ON_DATAMAP_DEFAULTVALUE).toBoolean
+      .getProperty(CarbonCommonConstants.VALIDATE_DIRECT_QUERY_ON_DATAMAP)
    var isThrowException = false
    // if validate query is enabled and relation contains pre aggregate data map


Change the comment according to the changes done

CarbonDataQA · 2018-12-27T10:23:05Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10212/

CarbonDataQA · 2018-12-27T12:52:11Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2233/

CarbonDataQA · 2018-12-27T20:32:22Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2260/

CarbonDataQA · 2018-12-28T00:39:55Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2055/

Shubh18s · 2018-12-28T03:04:55Z

retest this please

CarbonDataQA · 2018-12-28T04:14:25Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2262/

CarbonDataQA · 2018-12-28T04:24:29Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2057/

CarbonDataQA · 2018-12-28T05:44:53Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10309/

kunal642 · 2019-01-02T04:14:09Z

@Shubh18s Please fix 2.3 build

Shubh18s · 2019-01-02T05:56:07Z

retest this please

CarbonDataQA · 2019-01-02T06:09:15Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2107/

CarbonDataQA · 2019-01-02T07:19:42Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10361/

CarbonDataQA · 2019-01-02T07:29:45Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2312/

kumarvishal09 · 2019-01-02T08:24:49Z

core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

@@ -1445,16 +1445,10 @@ private CarbonCommonConstants() {

  @CarbonProperty(dynamicConfigurable = true)
  public static final String SUPPORT_DIRECT_QUERY_ON_DATAMAP =
-      "carbon.query.directQueryOnDataMap.enabled";
+          "carbon.query.directQueryOnDataMap.enabled";


Revert this change

kumarvishal09 · 2019-01-02T08:25:31Z

...2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala

-        aggregateColumns += s"${a.getAggFunction match {
-          case "count" => "sum"
-          case _ => a.getAggFunction}}(${a.getColumnName})"
+        aggregateColumns += s"${


revert this change

kumarvishal09 · 2019-01-02T08:27:46Z

...2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala

  }

-  /**
+  /** CarbonProperties.getInstance().addProperty(CarbonCommonConstants


why this change is requried??

CarbonDataQA · 2019-01-02T14:11:21Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2125/

CarbonDataQA · 2019-01-02T14:47:01Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2331/

CarbonDataQA · 2019-01-02T14:57:07Z

Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10379/

Shubh18s · 2019-01-03T05:22:11Z

retest this please

CarbonDataQA · 2019-01-03T05:34:13Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2136/

CarbonDataQA · 2019-01-03T06:44:59Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10390/

CarbonDataQA · 2019-01-03T06:59:17Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2342/

CarbonDataQA · 2019-01-03T11:59:41Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2358/

CarbonDataQA · 2019-01-03T12:40:19Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2152/

CarbonDataQA · 2019-01-03T13:22:01Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10404/

kumarvishal09 · 2019-01-04T11:10:13Z

retest this please

CarbonDataQA · 2019-01-04T12:15:15Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2168/

CarbonDataQA · 2019-01-04T12:41:04Z

Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10421/

CarbonDataQA · 2019-01-04T12:48:43Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2382/

Shubh18s · 2019-01-07T04:37:49Z

Respected Reviewer please Merge this PR.

kumarvishal09 · 2019-01-07T08:45:19Z

LGTM

xubo245 · 2019-01-16T14:24:13Z

Why haven't test case for this PR? @Shubh18s @kumarvishal09

Problem - Load and Select query was failing on table with preaggregate datamap. Cause - Previously if query on datamap was not enabled in thread, there was no check afterwards. Solution - After checking whether thread param for Direct Query On Datamap is enable. If not enable, we check in session params and then global. This closes #3010

Problem - Load and Select query was failing on table with preaggregate datamap. Cause - Previously if query on datamap was not enabled in thread, there was no check afterwards. Solution - After checking whether thread param for Direct Query On Datamap is enable. If not enable, we check in session params and then global. This closes apache#3010

Shubh18s changed the title ~~Fix PreAggregate Datamap Issue~~ [CARBONDATA-3189] Fix PreAggregate Datamap Issue Dec 20, 2018

Shubh18s closed this Dec 21, 2018

Shubh18s reopened this Dec 21, 2018

kunal642 suggested changes Dec 27, 2018

View reviewed changes

Shubh18s force-pushed the PreAgg branch from 1448459 to 349c912 Compare December 27, 2018 09:33

Shubh18s force-pushed the PreAgg branch 2 times, most recently from 9a4df75 to 80094ca Compare December 27, 2018 14:10

kumarvishal09 reviewed Jan 2, 2019

View reviewed changes

Shubh18s force-pushed the PreAgg branch from 80094ca to fcb24c1 Compare January 2, 2019 13:05

Fix PreAggregate Datamap issue

2656188

Shubh18s force-pushed the PreAgg branch from fcb24c1 to 2656188 Compare January 3, 2019 09:41

asfgit closed this in aad9aab Jan 7, 2019

[CARBONDATA-3189] Fix PreAggregate Datamap Issue #3010

[CARBONDATA-3189] Fix PreAggregate Datamap Issue #3010

Conversation

Shubh18s commented Dec 20, 2018

CarbonDataQA commented Dec 20, 2018

jackylk commented Dec 20, 2018

CarbonDataQA commented Dec 20, 2018

CarbonDataQA commented Dec 20, 2018

qiuchenjian commented Dec 21, 2018

Shubh18s commented Dec 21, 2018

brijoobopanna commented Dec 24, 2018

brijoobopanna commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

CarbonDataQA commented Dec 24, 2018

kumarvishal09 commented Dec 24, 2018

CarbonDataQA commented Dec 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 27, 2018

CarbonDataQA commented Dec 28, 2018

Shubh18s commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

CarbonDataQA commented Dec 28, 2018

kunal642 commented Jan 2, 2019

Shubh18s commented Jan 2, 2019

CarbonDataQA commented Jan 2, 2019

CarbonDataQA commented Jan 2, 2019

CarbonDataQA commented Jan 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Jan 2, 2019

CarbonDataQA commented Jan 2, 2019

CarbonDataQA commented Jan 2, 2019

Shubh18s commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

CarbonDataQA commented Jan 3, 2019

kumarvishal09 commented Jan 4, 2019

CarbonDataQA commented Jan 4, 2019

CarbonDataQA commented Jan 4, 2019

CarbonDataQA commented Jan 4, 2019

Shubh18s commented Jan 7, 2019

kumarvishal09 commented Jan 7, 2019

xubo245 commented Jan 16, 2019 • edited

xubo245 commented Jan 16, 2019 •

edited