[CARBONDATA-3689] Support independent MV extension and MV syntax #3609

jackylk · 2020-02-10T14:30:21Z

Why is this PR needed?

To better promote Materialized View usage, we can make Materialized View as an independent extension for Apache Spark

What changes were proposed in this PR?

This PR adds Materialized View extension for spark:
Following SQL command are supported:

CREATE MATERIALIZED VIEW
DROP MATERIALIZED VIEW
SHOW MATERIALIZED VIEW
REFRESH MATERIALIZED VIEW

Following optimizer rules are added:

Rewrite SQL statement by matching existing MV and
select the lowest cost MV

Does this PR introduce any user interface change?

Yes. (new SQL syntax is added)

Is any new testcase added?

No (existing Datamap testcase is modified to use MV syntax)

CarbonDataQA1 · 2020-02-10T14:45:32Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/215/

CarbonDataQA1 · 2020-02-10T14:52:39Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1917/

CarbonDataQA1 · 2020-02-10T15:30:22Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/218/

CarbonDataQA1 · 2020-02-10T16:15:30Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/220/

CarbonDataQA1 · 2020-02-10T17:09:30Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1922/

jackylk · 2020-02-11T01:39:03Z

retest this please

CarbonDataQA1 · 2020-02-11T02:00:25Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/223/

CarbonDataQA1 · 2020-02-11T02:58:21Z

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1925/

jackylk · 2020-02-11T12:26:28Z

retest this please

CarbonDataQA1 · 2020-02-11T12:46:03Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/238/

CarbonDataQA1 · 2020-02-11T13:42:11Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1940/

Indhumathi27 · 2020-02-12T08:54:50Z

datamap/mv/core/src/test/scala/org/apache/carbondata/mv/testutil/Tpcds_1_4_QueryBatch.scala

- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *


why License is removed?

CarbonDataQA1 · 2020-02-12T10:16:24Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/256/

CarbonDataQA1 · 2020-02-12T11:13:15Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1959/

Indhumathi27 · 2020-02-13T06:45:43Z

datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVParser.scala

+   * MVPROPERTIES('KEY'='VALUE') AS mv_query_statement
+   */
+  private lazy val createMV: Parser[LogicalPlan] =
+    CREATE ~> MATERIALIZED ~> VIEW ~> opt(IF ~> NOT ~> EXISTS) ~ ident ~


Suggested change

CREATE ~> MATERIALIZED ~> VIEW ~> opt(IF ~> NOT ~> EXISTS) ~ ident ~

CREATE ~> MATERIALIZED ~> VIEW ~> opt(IF ~> NOT ~> EXISTS) ~ ident ~ ontable ~(ident <~ ".").? ~ ident-

i suggest db.table name can be provided while creating materilaised views, which will be helpful in future.

On table is not required because MV can be a join, so I think better not forcing user to give this.
Is there any helpful scenario for this?

for supporting multi-tenant, it can be useful in creating mv under a specified database in create command

For supporting multi-tenant, I think the mv table should be created under current database, just like a create normal table does. What do you think?

what if user creates mv from different database, where database's involved in query is different.

Indhumathi27 · 2020-02-13T06:46:26Z

datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVParser.scala

+   * DROP MATERIALIZED VIEW IF EXISTS mv_name
+   */
+  private lazy val dropMV: Parser[LogicalPlan] =
+    DROP ~> MATERIALIZED ~> VIEW ~> opt(IF ~> EXISTS) ~ ident <~ opt(";") ^^ {


Suggested change

DROP ~> MATERIALIZED ~> VIEW ~> opt(IF ~> EXISTS) ~ ident <~ opt(";") ^^ {

DROP ~> MATERIALIZED ~> VIEW ~> opt(IF ~> EXISTS) ~ ident ~ONTABLE ~ (ident <~ ".").? ~ ident<~ opt(";") ^^ {

On table is not required because MV can be a join, so I think better not forcing user to give this

Indhumathi27 · 2020-02-13T06:48:11Z

datamap/mv/core/src/main/scala/org/apache/carbondata/mv/extension/MVParser.scala

+   * REBUILD MATERIALIZED VIEW mv_name
+   */
+  private lazy val rebuildMV: Parser[LogicalPlan] =
+    REBUILD ~> MATERIALIZED ~> VIEW ~> ident <~ opt(";") ^^ {


Suggested change

REBUILD ~> MATERIALIZED ~> VIEW ~> ident <~ opt(";") ^^ {

REBUILD ~> MATERIALIZED ~> VIEW ~> ident ~ ONTABLE ~ (ident <~ ".").? ~ ident<~ opt(";") ^^ {

On table is not required because MV can be a join, so I think better not forcing user to give this

CarbonDataQA1 · 2020-02-14T08:06:49Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/282/

CarbonDataQA1 · 2020-02-14T09:28:28Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1985/

CarbonDataQA1 · 2020-02-14T10:22:10Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/286/

CarbonDataQA1 · 2020-02-14T11:43:24Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1989/

CarbonDataQA1 · 2020-02-14T12:47:49Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/290/

CarbonDataQA1 · 2020-02-14T14:09:19Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1994/

CarbonDataQA1 · 2020-02-15T04:46:47Z

Build Success with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/300/

CarbonDataQA1 · 2020-02-15T06:04:21Z

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/2004/

jackylk changed the title ~~[WIP] Support MV extension and MV syntax~~ [CARBONDATA-3689] Support independent MV extension and MV syntax Feb 11, 2020

jackylk mentioned this pull request Feb 11, 2020

[CARBONDATA-3694] Separate Materialized View command from DataMap command #3612

Closed

Indhumathi27 reviewed Feb 12, 2020

View reviewed changes

jackylk force-pushed the mv_extension branch from 55de52d to 94878c1 Compare February 12, 2020 09:56

Indhumathi27 reviewed Feb 13, 2020

View reviewed changes

jackylk force-pushed the mv_extension branch from 94878c1 to 484eecf Compare February 14, 2020 07:46

jackylk force-pushed the mv_extension branch from 484eecf to cac1c8b Compare February 14, 2020 10:01

jackylk force-pushed the mv_extension branch from cac1c8b to 03c4f3e Compare February 14, 2020 12:25

add MV extension

24cc532

jackylk force-pushed the mv_extension branch from 03c4f3e to 24cc532 Compare February 15, 2020 04:26

jackylk closed this Feb 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CARBONDATA-3689] Support independent MV extension and MV syntax #3609

[CARBONDATA-3689] Support independent MV extension and MV syntax #3609

jackylk commented Feb 10, 2020 •

edited

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

jackylk commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

jackylk commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

Indhumathi27 Feb 12, 2020

jackylk Feb 12, 2020

CarbonDataQA1 commented Feb 12, 2020

CarbonDataQA1 commented Feb 12, 2020

Indhumathi27 Feb 13, 2020

jackylk Feb 13, 2020

Indhumathi27 Feb 14, 2020

jackylk Feb 14, 2020

Indhumathi27 Feb 14, 2020 •

edited

Indhumathi27 Feb 13, 2020

jackylk Feb 13, 2020

Indhumathi27 Feb 13, 2020

jackylk Feb 13, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 15, 2020

CarbonDataQA1 commented Feb 15, 2020

	CREATE ~> MATERIALIZED ~> VIEW ~> opt(IF ~> NOT ~> EXISTS) ~ ident ~
	CREATE ~> MATERIALIZED ~> VIEW ~> opt(IF ~> NOT ~> EXISTS) ~ ident ~ ontable ~(ident <~ ".").? ~ ident-

	DROP ~> MATERIALIZED ~> VIEW ~> opt(IF ~> EXISTS) ~ ident <~ opt(";") ^^ {
	DROP ~> MATERIALIZED ~> VIEW ~> opt(IF ~> EXISTS) ~ ident ~ONTABLE ~ (ident <~ ".").? ~ ident<~ opt(";") ^^ {

	REBUILD ~> MATERIALIZED ~> VIEW ~> ident <~ opt(";") ^^ {
	REBUILD ~> MATERIALIZED ~> VIEW ~> ident ~ ONTABLE ~ (ident <~ ".").? ~ ident<~ opt(";") ^^ {

[CARBONDATA-3689] Support independent MV extension and MV syntax #3609

[CARBONDATA-3689] Support independent MV extension and MV syntax #3609

Conversation

jackylk commented Feb 10, 2020 • edited

Why is this PR needed?

What changes were proposed in this PR?

Does this PR introduce any user interface change?

Is any new testcase added?

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

CarbonDataQA1 commented Feb 10, 2020

jackylk commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

jackylk commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

CarbonDataQA1 commented Feb 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Feb 12, 2020

CarbonDataQA1 commented Feb 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Indhumathi27 Feb 14, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 14, 2020

CarbonDataQA1 commented Feb 15, 2020

CarbonDataQA1 commented Feb 15, 2020

jackylk commented Feb 10, 2020 •

edited

Indhumathi27 Feb 14, 2020 •

edited