[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to create table #2078

xubo245 · 2018-03-19T12:02:02Z

Support the syntax of 'STORED AS CARBONDATA' to create table, for example:
sql("CREATE TABLE carbon_table(key INT, value STRING) STORED AS CARBONDATA")
Improving compatibility with Hive and giving better user experience

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
Any backward compatibility impacted?
Document update required?
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

CarbonDataQA · 2018-03-19T12:06:56Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3193/

CarbonDataQA · 2018-03-19T12:10:05Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4427/

ravipesala · 2018-03-19T12:47:49Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3948/

CarbonDataQA · 2018-03-20T02:23:30Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3202/

CarbonDataQA · 2018-03-20T02:25:59Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4435/

ravipesala · 2018-03-20T02:26:36Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3956/

CarbonDataQA · 2018-03-20T07:23:29Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3206/

CarbonDataQA · 2018-03-20T07:26:42Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4439/

ravipesala · 2018-03-20T07:32:23Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3961/

CarbonDataQA · 2018-03-20T10:23:44Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4449/

CarbonDataQA · 2018-03-20T10:28:07Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3216/

xubo245 · 2018-03-20T10:38:36Z

retest this please

CarbonDataQA · 2018-03-20T11:43:21Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3219/

CarbonDataQA · 2018-03-20T11:51:24Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4452/

ravipesala · 2018-03-20T12:39:47Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3968/

ravipesala · 2018-03-20T14:01:44Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3970/

xubo245 · 2018-03-21T01:21:36Z

@jackylk Please review it.

jackylk · 2018-03-21T11:22:41Z

integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSparkSqlParser.scala

@@ -91,6 +91,9 @@ class CarbonHelperSqlAstBuilder(conf: SQLConf,
      case Some(value) =>
        if (value.children.get(1).getText.equalsIgnoreCase("by")) {
          value.storageHandler().STRING().getSymbol.getText
+        } else if (value.children.get(1).getText.equalsIgnoreCase("as")


can you call value.children.get(1).getText once and use the result

jackylk · 2018-03-21T11:24:33Z

integration/spark2/src/main/spark2.1/org/apache/spark/sql/hive/CarbonSessionState.scala

@@ -326,7 +326,8 @@ class CarbonSqlAstBuilder(conf: SQLConf, parser: CarbonSpark2SqlParser, sparkSes
    val fileStorage = helper.getFileStorage(ctx.createFileFormat)

    if (fileStorage.equalsIgnoreCase("'carbondata'") ||
-        fileStorage.equalsIgnoreCase("'org.apache.carbondata.format'")) {
+      fileStorage.equalsIgnoreCase("carbondata") ||


incorrect identation

line 329 is repeated

It's different: 'carbondata' and carbondata.

carbon supported syntax "stored by 'carbondata'" before, but didn't supported syntax "stored by carbondata".
we need support syntax "STORED AS carbondata" like "STORED AS parquet"

stored by carbondata is not required

jackylk · 2018-03-21T11:26:25Z

integration/spark2/src/main/spark2.2/org/apache/spark/sql/hive/CarbonSessionState.scala

@@ -325,7 +325,8 @@ class CarbonSqlAstBuilder(conf: SQLConf, parser: CarbonSpark2SqlParser, sparkSes
    val fileStorage = helper.getFileStorage(ctx.createFileFormat)

    if (fileStorage.equalsIgnoreCase("'carbondata'") ||
-        fileStorage.equalsIgnoreCase("'org.apache.carbondata.format'")) {
+      fileStorage.equalsIgnoreCase("carbondata") ||


repeated logic

It's different: 'carbondata' and carbondata.

jackylk · 2018-03-21T11:27:16Z

integration/spark2/src/test/scala/org/apache/spark/SparkCommandSuite.scala

+
+  test("CARBONDATA-734: Don't Support the syntax of 'STORED BY 'PARQUET''") {
+    try {
+      sql("CREATE TABLE src_pqt(key INT, value STRING) STORED AS 'PARQUET'")


This is not correct, stored as parquet or orc should be supported

Now carbon support syntax "stored as parquet", but don't support syntax "stored as 'parquet'".

Whether do we support syntax "stored as 'parquet'"?

CarbonDataQA · 2018-03-22T02:35:40Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3263/

CarbonDataQA · 2018-03-22T02:37:17Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4492/

ravipesala · 2018-03-22T03:07:17Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4004/

xubo245 · 2018-03-22T09:41:28Z

retest this please

CarbonDataQA · 2018-03-22T10:24:59Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4504/

CarbonDataQA · 2018-03-22T10:36:35Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3275/

CarbonDataQA · 2018-03-22T14:11:55Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4510/

CarbonDataQA · 2018-03-22T14:15:51Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3282/

ravipesala · 2018-03-22T15:59:33Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4016/

xubo245 · 2018-03-23T01:15:59Z

retest this please

xubo245 · 2018-03-23T01:16:08Z

retest sdv please

CarbonDataQA · 2018-03-23T01:54:07Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4513/

CarbonDataQA · 2018-03-23T01:57:38Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3285/

ravipesala · 2018-03-23T02:02:22Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4018/

xubo245 · 2018-03-23T08:52:27Z

retest this please

xubo245 · 2018-03-23T08:52:33Z

retest sdv please

CarbonDataQA · 2018-03-23T09:33:10Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3299/

CarbonDataQA · 2018-03-23T09:34:48Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4527/

ravipesala · 2018-03-23T10:44:37Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4027/

…ate table fix scalastyle fix spark2.1 error optimize optimize optimize according review comment optimize

CarbonDataQA · 2018-03-26T02:02:12Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4583/

CarbonDataQA · 2018-03-26T02:03:50Z

Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3357/

ravipesala · 2018-03-26T02:08:37Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4079/

xubo245 · 2018-03-26T02:31:34Z

retest sdv please

ravipesala · 2018-03-26T03:58:13Z

SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4081/

jackylk · 2018-03-26T15:18:51Z

LGTM

…ate table This closes apache#2078

xubo245 changed the title ~~[WIP][CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to create table~~ [CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to create table Mar 20, 2018

jackylk reviewed Mar 21, 2018

View reviewed changes

[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to cre…

67b4615

…ate table fix scalastyle fix spark2.1 error optimize optimize optimize according review comment optimize

xubo245 force-pushed the CARBONDATA-2262UsingAndStoredAs branch from f7d203a to 67b4615 Compare March 26, 2018 01:20

asfgit closed this in ee9df2e Mar 26, 2018

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Apr 4, 2018

[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to cre…

e828140

…ate table This closes apache#2078

xubo245 mentioned this pull request Apr 27, 2018

[CARBONDATA-2404] Add documentation for using carbondata and stored as carbondata #2230

Closed

5 tasks

anubhav100 pushed a commit to anubhav100/incubator-carbondata that referenced this pull request Jun 22, 2018

[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to cre…

1ec4c5b

…ate table This closes apache#2078

[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to create table #2078

[CARBONDATA-2262] Support the syntax of 'STORED AS CARBONDATA' to create table #2078

Conversation

xubo245 commented Mar 19, 2018 • edited

CarbonDataQA commented Mar 19, 2018

CarbonDataQA commented Mar 19, 2018

ravipesala commented Mar 19, 2018

CarbonDataQA commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

ravipesala commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

ravipesala commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

xubo245 commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

CarbonDataQA commented Mar 20, 2018

ravipesala commented Mar 20, 2018

ravipesala commented Mar 20, 2018

xubo245 commented Mar 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xubo245 Mar 22, 2018 • edited

Choose a reason for hiding this comment

CarbonDataQA commented Mar 22, 2018

CarbonDataQA commented Mar 22, 2018

ravipesala commented Mar 22, 2018

xubo245 commented Mar 22, 2018

CarbonDataQA commented Mar 22, 2018

CarbonDataQA commented Mar 22, 2018

CarbonDataQA commented Mar 22, 2018

CarbonDataQA commented Mar 22, 2018

ravipesala commented Mar 22, 2018

xubo245 commented Mar 23, 2018

xubo245 commented Mar 23, 2018

CarbonDataQA commented Mar 23, 2018

CarbonDataQA commented Mar 23, 2018

ravipesala commented Mar 23, 2018

xubo245 commented Mar 23, 2018

xubo245 commented Mar 23, 2018

CarbonDataQA commented Mar 23, 2018

CarbonDataQA commented Mar 23, 2018

ravipesala commented Mar 23, 2018

CarbonDataQA commented Mar 26, 2018

CarbonDataQA commented Mar 26, 2018

ravipesala commented Mar 26, 2018

xubo245 commented Mar 26, 2018

ravipesala commented Mar 26, 2018

jackylk commented Mar 26, 2018

xubo245 commented Mar 19, 2018 •

edited

xubo245 Mar 22, 2018 •

edited