[CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate table #1508

kunal642 · 2017-11-16T12:35:22Z

Block load/insert on pre-aggregate table

Any interfaces changed?
No
Any backward compatibility impacted?
No
Document update required?
No
Testing done
Added test case
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

gvramana · 2017-11-16T13:05:51Z

integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala

@@ -212,6 +212,18 @@ object CarbonSession {
    ThreadLocalSessionInfo.setCarbonSessionInfo(currentThreadSessionInfo)
  }

+  def threadUnset(key: String): Unit = {
+    var currentThreadSessionInfo = ThreadLocalSessionInfo.getCarbonSessionInfo
+    if (currentThreadSessionInfo == null) {


If null then not required to unset

gvramana · 2017-11-16T13:10:52Z

.../main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala

@@ -51,11 +51,39 @@ object LoadPostAggregateListener extends OperationEventListener {
        val childDatabaseName = dataMapSchema.getRelationIdentifier.getDatabaseName
        val selectQuery = dataMapSchema.getProperties.get("CHILD_SELECT QUERY")
        sparkSession.sql(s"insert into $childDatabaseName.$childTableName $selectQuery")
+        CarbonSession.threadUnset(CarbonCommonConstants.CARBON_INPUT_SEGMENTS +


add this unset to finally

gvramana · 2017-11-16T13:18:06Z

.../main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala

+    val carbonLoadModel = loadEvent.carbonLoadModel
+    val table = carbonLoadModel.getCarbonDataLoadSchema.getCarbonTable
+    val carbonSessionInfo = ThreadLocalSessionInfo.getCarbonSessionInfo
+    val isInternalLoadCall = carbonSessionInfo.getSessionParams.getAll.keySet().asScala


Use specifc parameter instead of using this VALIDATE_CARBON_INPUT_SEGMENTS

CarbonDataQA · 2017-11-16T13:59:42Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1182/

CarbonDataQA · 2017-11-17T05:50:10Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1210/

CarbonDataQA · 2017-11-17T07:51:48Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1215/

jackylk · 2017-11-17T09:29:34Z

core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java

+   * Whether load/insert command is fired internally or by the user.
+   * Used to block load/insert on pre-aggregate if fired by user
+   */
+  public static final String IS_INTERNAL_LOAD_CALL = "is.internal.load.call";


Seems no testcase for this option. And the option name should start with carbon

This option/property will not be exposed to the user. It will be set by the post load listener to know whether the load is fired by the user or is it an internal call.

Test case is added in TestPreAggregateLoad

kunal642 · 2017-11-17T14:52:37Z

retest this please

CarbonDataQA · 2017-11-17T16:38:08Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1240/

CarbonDataQA · 2017-11-19T14:27:57Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1288/

kunal642 · 2017-11-19T14:29:08Z

retest this please

CarbonDataQA · 2017-11-19T14:56:01Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1289/

CarbonDataQA · 2017-11-20T08:09:25Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1308/

CarbonDataQA · 2017-11-20T10:23:33Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1313/

CarbonDataQA · 2017-11-21T05:04:43Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1324/

CarbonDataQA · 2017-11-21T06:25:51Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1326/

ravipesala · 2017-11-21T07:28:33Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1784/

kunal642 · 2017-11-21T08:45:12Z

@ravipesala Please review

ravipesala · 2017-11-22T01:44:37Z

retest sdv please

ravipesala · 2017-11-22T01:54:54Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1801/

ravipesala · 2017-11-22T01:57:27Z

retest sdv please

ravipesala · 2017-11-29T08:49:07Z

...la/org/apache/spark/sql/execution/command/preaaggregate/CreatePreAggregateTableCommand.scala

-          s"insert into ${ tableModel.databaseName }.${ tableModel.tableName } $queryString")
+      val availableLoads = PreAggregateUtil.checkMainTableLoad(parentTable)
+      if (availableLoads) {
+        val headers = childSchema.getChildSchema.getListOfColumns.asScala.map(_.getColumnName)


Move line done

ravipesala · 2017-11-29T08:49:34Z

...la/org/apache/spark/sql/execution/command/preaaggregate/CreatePreAggregateTableCommand.scala

+      if (availableLoads) {
+        val headers = childSchema.getChildSchema.getListOfColumns.asScala.map(_.getColumnName)
+          .mkString(",")
+        val childDataFrame = sparkSession.sql(new CarbonSpark2SqlParser()


Move down after (

ravipesala · 2017-11-29T08:50:41Z

.../main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala

+        val childDataFrame = sparkSession.sql(new CarbonSpark2SqlParser()
+          .addPreAggLoadFunction(s"${ dataMapSchema.getProperties.get("CHILD_SELECT QUERY") } "))
+          .drop("preAggLoad")
+        val headers = dataMapSchema.getChildSchema.getListOfColumns.asScala.map(_.getColumnName)


ravipesala · 2017-11-29T08:52:40Z

.../main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateListeners.scala

+      throw new UnsupportedOperationException(
+        "Cannot insert/load data directly into pre-aggregate table")
+    }
+


remove line

ravipesala · 2017-11-29T08:53:34Z

...2/src/main/scala/org/apache/spark/sql/execution/command/preaaggregate/PreAggregateUtil.scala

@@ -253,8 +260,8 @@ object PreAggregateUtil {
          carbonTable.getColumnByName(parentTableName, attr.name).getColumnId,
          parentTableName,
          parentDatabaseName, parentTableId = parentTableId)
-      case _ =>
-        throw new MalformedCarbonCommandException("Un-Supported Aggregation Type")
+      case a@_ =>


Keep as case others

ravipesala · 2017-11-29T08:55:09Z

integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonPreAggregateRules.scala

+                  attr.name + "_sum")(),
+                  Alias(attrExpression
+                    .copy(aggregateFunction = Count(cast),
+                      resultId = NamedExpression.newExprId), attr.name + "_count")())


Please format it properly

ravipesala · 2017-11-29T08:55:42Z

integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonPreAggregateRules.scala

+    filteredExpressions
+      .exists {
+        expr => !expr.name.equalsIgnoreCase("PreAgg") &&
+                        expr.name.equalsIgnoreCase("preAggLoad")


format properly

ravipesala · 2017-11-29T08:57:38Z

integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala

+  def addPreAggLoadFunction(sql: String): String = {
+    addPreAggLoad(new lexical.Scanner(sql.toLowerCase)) match {
+      case Success(query, _) => query
+      case _ => throw new MalformedCarbonCommandException(


move down line

CarbonDataQA · 2017-12-02T15:03:06Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1639/

CarbonDataQA · 2017-12-02T15:54:04Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1641/

ravipesala · 2017-12-02T18:00:55Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2031/

kunal642 · 2017-12-03T07:12:21Z

@ravipesala Handled all the review comments

ravipesala · 2017-12-04T06:27:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2053/

CarbonDataQA · 2017-12-04T06:59:15Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1664/

ravipesala · 2017-12-04T07:07:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2055/

CarbonDataQA · 2017-12-04T08:21:01Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1668/

ravipesala · 2017-12-04T09:42:57Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2057/

CarbonDataQA · 2017-12-04T10:03:14Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1670/

CarbonDataQA · 2017-12-04T10:45:29Z

Build Success with Spark 2.2, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/400/

ravipesala · 2017-12-04T10:47:19Z

LGTM

…table This closes apache#1508

gvramana reviewed Nov 16, 2017

View reviewed changes

kunal642 force-pushed the block_load_on_preagg branch from 9d48d35 to 3848d85 Compare November 17, 2017 04:32

kunal642 force-pushed the block_load_on_preagg branch from 3848d85 to d2906e8 Compare November 17, 2017 06:51

jackylk reviewed Nov 17, 2017

View reviewed changes

kunal642 force-pushed the block_load_on_preagg branch from d2906e8 to c5f56c1 Compare November 19, 2017 13:46

kunal642 force-pushed the block_load_on_preagg branch 2 times, most recently from 955a4a8 to 718a9b1 Compare November 20, 2017 07:53

kunal642 force-pushed the block_load_on_preagg branch from 718a9b1 to 137496f Compare November 20, 2017 09:22

kunal642 force-pushed the block_load_on_preagg branch from 137496f to 70af7d8 Compare November 21, 2017 05:00

kunal642 force-pushed the block_load_on_preagg branch from 70af7d8 to 644b0fc Compare November 21, 2017 05:43

kunal642 changed the title ~~[CARBONDATA-1738] Block direct insert/load on pre-aggregate table~~ [CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate table Nov 21, 2017

ravipesala reviewed Nov 29, 2017

View reviewed changes

kunal642 force-pushed the block_load_on_preagg branch 2 times, most recently from 492575b to fa231e8 Compare December 2, 2017 14:56

kunal642 mentioned this pull request Dec 2, 2017

[CARBONDATA-1842]Fix 'wrong argument number' error of class Cast for Spark 2.2 when pattern matching #1600

Closed

5 tasks

kunal642 force-pushed the block_load_on_preagg branch from fa231e8 to b028824 Compare December 4, 2017 06:19

kunal642 force-pushed the block_load_on_preagg branch from b028824 to c1a8949 Compare December 4, 2017 06:56

blocked direct insert/load on pre-aggregate table

6e135d6

kunal642 force-pushed the block_load_on_preagg branch from c1a8949 to 6e135d6 Compare December 4, 2017 09:34

asfgit closed this in 5ae596b Dec 4, 2017

jatin9896 pushed a commit to jatin9896/incubator-carbondata that referenced this pull request Jan 5, 2018

[CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate …

39ec454

…table This closes apache#1508

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate …

dcc1e81

…table This closes apache#1508

[CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate table #1508

[CARBONDATA-1738] [PreAgg] Block direct insert/load on pre-aggregate table #1508

Conversation

kunal642 commented Nov 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Nov 16, 2017

CarbonDataQA commented Nov 17, 2017

CarbonDataQA commented Nov 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kunal642 commented Nov 17, 2017

CarbonDataQA commented Nov 17, 2017

CarbonDataQA commented Nov 19, 2017

kunal642 commented Nov 19, 2017

CarbonDataQA commented Nov 19, 2017

CarbonDataQA commented Nov 20, 2017

CarbonDataQA commented Nov 20, 2017

CarbonDataQA commented Nov 21, 2017

CarbonDataQA commented Nov 21, 2017

ravipesala commented Nov 21, 2017

kunal642 commented Nov 21, 2017

ravipesala commented Nov 22, 2017

ravipesala commented Nov 22, 2017

ravipesala commented Nov 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA commented Dec 2, 2017

CarbonDataQA commented Dec 2, 2017

ravipesala commented Dec 2, 2017

kunal642 commented Dec 3, 2017

ravipesala commented Dec 4, 2017

CarbonDataQA commented Dec 4, 2017

ravipesala commented Dec 4, 2017

CarbonDataQA commented Dec 4, 2017

ravipesala commented Dec 4, 2017

CarbonDataQA commented Dec 4, 2017

CarbonDataQA commented Dec 4, 2017

ravipesala commented Dec 4, 2017