[CARBONDATA-2883][ExternalFormat] block some operations on external format table #2656

Sssan520 · 2018-08-24T03:18:40Z

Now some operatons are not supported for external format table, so those operations will be blocked as bellows:
1. update data on external format table
2. delete data on external format table
3. add columns on external format table
4. drop columns on external format table
5. change column datatype on external format table

The operation in following scenario is not allowed:
create external format table if table columns not in tblproperty '{format}.header'

And add external format information for command 'desc formatted table'.
e.g.
CREATE TABLE fact_carbon_csv_table(empname String, empno int, designation string)
STORED BY 'carbondata'
TBLPROPERTIES(
'format'='csv',
'csv.header'='empno, empname, designation, salary',
'csv.delimiter'='|'
)
desc formatted fact_carbon_csv_table:
|##Detailed External-Format Information| |
|format |csv |
|csv.delimiter || |
|csv.header |empno,empname,designation,salary |

Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:

Any interfaces changed?
No
Any backward compatibility impacted?
No
Document update required?
No
Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
has added test case.
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA

CarbonDataQA · 2018-08-24T03:21:09Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6739/

CarbonDataQA · 2018-08-24T03:23:39Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8016/

ravipesala · 2018-08-24T03:32:58Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6365/

Sssan520 · 2018-08-24T03:57:42Z

retest this please

CarbonDataQA · 2018-08-24T04:00:38Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6741/

CarbonDataQA · 2018-08-24T04:03:37Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8018/

xuchuanyin · 2018-08-24T10:32:43Z

...st/scala/org/apache/carbondata/spark/testsuite/externalformat/CsvBasedCarbonTableSuite.scala

+  }
+
+  test("test desc formatted for external format table") {
+    sql(


better to extract the createTableCommand to a function to reduce duplicated code

xuchuanyin · 2018-08-24T10:33:23Z

...in/scala/org/apache/spark/sql/execution/command/mutation/CarbonProjectForDeleteCommand.scala

@@ -49,7 +49,9 @@ private[sql] case class CarbonProjectForDeleteCommand(
    if (!carbonTable.getTableInfo.isTransactionalTable) {
      throw new MalformedCarbonCommandException("Unsupported operation on non transactional table")
    }
-
+    if (carbonTable.isExternalFormat) {
+      throw new MalformedCarbonCommandException("Unsupported operation on external table")


'on external format table', not 'on external table'

xuchuanyin · 2018-08-24T10:34:15Z

core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java

+   * if format is 'carbondata' or '' or null, return false
+   * else return true.
+   */
+  public boolean isExternalFormat() {


please optimize it to 'isExternalFormatTable'

Sssan520 · 2018-08-24T12:02:17Z

retest this please

ravipesala · 2018-08-24T12:19:42Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6390/

CarbonDataQA · 2018-08-24T12:44:21Z

Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6766/

CarbonDataQA · 2018-08-24T12:45:37Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8044/

xuchuanyin · 2018-08-25T03:21:46Z

...st/scala/org/apache/carbondata/spark/testsuite/externalformat/CsvBasedCarbonTableSuite.scala

+    val index = result.indexWhere(_.get(0).toString.
+      contains("##Detailed External-Format Information"))
+    assert(index > -1)
+    assert(result(index + 1).get(0).toString.contains("format"))


the sequence of the content is not ensured, it may change in the future.
You can use checkExistence to check whether the keywords are there in the DataFrame

xuchuanyin · 2018-08-25T03:22:54Z

In addition, I think you can rename the test suite to 'CsvExternalFormatBasicFunctionSuite'

CarbonDataQA · 2018-08-25T07:32:18Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8061/

ravipesala · 2018-08-25T07:46:38Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6403/

CarbonDataQA · 2018-08-27T02:44:33Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8078/

CarbonDataQA · 2018-08-27T02:58:55Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8079/

CarbonDataQA · 2018-08-27T04:25:59Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/20/

CarbonDataQA · 2018-08-28T06:44:04Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8106/

ravipesala · 2018-08-28T07:26:41Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6436/

CarbonDataQA · 2018-08-28T07:39:22Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8109/

CarbonDataQA · 2018-08-28T08:01:19Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8111/

…ormat table

ravipesala · 2018-08-28T08:41:59Z

SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6439/

Sssan520 · 2018-08-28T08:45:49Z

retest this please

Sssan520 · 2018-08-28T09:07:48Z

retest this please

CarbonDataQA · 2018-08-28T09:23:28Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8117/

CarbonDataQA · 2018-08-28T09:58:31Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/43/

Sssan520 · 2018-08-28T10:28:03Z

retest this please

CarbonDataQA · 2018-08-28T10:35:09Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8118/

Sssan520 · 2018-08-28T10:41:02Z

retest this please

CarbonDataQA · 2018-08-28T11:24:27Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/45/

CarbonDataQA · 2018-08-28T11:47:51Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8119/

Sssan520 · 2018-08-28T12:31:12Z

retest this please

CarbonDataQA · 2018-08-28T13:12:27Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/47/

CarbonDataQA · 2018-08-28T13:38:23Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/54/

CarbonDataQA · 2018-08-28T13:47:58Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8123/

Sssan520 · 2018-08-29T01:12:06Z

retest this please

CarbonDataQA · 2018-08-29T02:22:51Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8130/

CarbonDataQA · 2018-08-29T02:25:03Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/59/

Sssan520 · 2018-08-29T02:53:12Z

retest this please

CarbonDataQA · 2018-08-29T04:02:46Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8133/

CarbonDataQA · 2018-08-29T04:05:54Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/62/

Sssan520 · 2018-08-29T06:39:32Z

retest this please

CarbonDataQA · 2018-08-29T07:46:05Z

Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8135/

CarbonDataQA · 2018-08-29T08:13:47Z

Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/64/

xuchuanyin · 2018-08-29T12:40:23Z

retest this please

CarbonDataQA · 2018-08-29T13:50:56Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/8146/

CarbonDataQA · 2018-08-29T14:12:59Z

Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/75/

CarbonDataQA · 2018-09-05T14:57:31Z

Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/20/

CarbonDataQA · 2019-07-24T09:07:16Z

Build Failed with Spark 2.1.0, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.1/63/

CarbonDataQA · 2019-07-24T09:37:47Z

Build Failed with Spark 2.3.2, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/62/

CarbonDataQA1 · 2020-02-03T07:03:45Z

Build Failed with Spark 2.4.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.4/63/

Sssan520 changed the title ~~[ExternalFormat] block some operations on external format table~~ [CARBONDATA-2883][ExternalFormat] block some operations on external format table Aug 24, 2018

xuchuanyin reviewed Aug 24, 2018

View reviewed changes

Sssan520 force-pushed the external-format-ddl branch from 3b501f9 to f68f19a Compare August 24, 2018 12:00

xuchuanyin reviewed Aug 25, 2018

View reviewed changes

Sssan520 force-pushed the external-format-ddl branch from f68f19a to 05b4d02 Compare August 25, 2018 07:30

Sssan520 force-pushed the external-format-ddl branch from 05b4d02 to 167e649 Compare August 28, 2018 07:11

[CARBONDATA-2883][ExternalFormat] block some operations on external f…

5b6a19c

…ormat table

Sssan520 force-pushed the external-format-ddl branch from 167e649 to 5b6a19c Compare August 28, 2018 08:25

[CARBONDATA-2883][ExternalFormat] block some operations on external format table #2656

Are you sure you want to change the base?

[CARBONDATA-2883][ExternalFormat] block some operations on external format table #2656

Conversation

Sssan520 commented Aug 24, 2018 • edited

CarbonDataQA commented Aug 24, 2018

CarbonDataQA commented Aug 24, 2018

ravipesala commented Aug 24, 2018

Sssan520 commented Aug 24, 2018

CarbonDataQA commented Aug 24, 2018

CarbonDataQA commented Aug 24, 2018

xuchuanyin Aug 24, 2018

Choose a reason for hiding this comment

Sssan520 Aug 24, 2018

Choose a reason for hiding this comment

xuchuanyin Aug 24, 2018

Choose a reason for hiding this comment

Sssan520 Aug 24, 2018

Choose a reason for hiding this comment

xuchuanyin Aug 24, 2018

Choose a reason for hiding this comment

Sssan520 Aug 24, 2018

Choose a reason for hiding this comment

Sssan520 commented Aug 24, 2018

ravipesala commented Aug 24, 2018

CarbonDataQA commented Aug 24, 2018

CarbonDataQA commented Aug 24, 2018

xuchuanyin Aug 25, 2018

Choose a reason for hiding this comment

xuchuanyin commented Aug 25, 2018

CarbonDataQA commented Aug 25, 2018

ravipesala commented Aug 25, 2018

CarbonDataQA commented Aug 27, 2018

CarbonDataQA commented Aug 27, 2018

CarbonDataQA commented Aug 27, 2018

CarbonDataQA commented Aug 28, 2018

ravipesala commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

ravipesala commented Aug 28, 2018

Sssan520 commented Aug 28, 2018

Sssan520 commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

Sssan520 commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

Sssan520 commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

Sssan520 commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

CarbonDataQA commented Aug 28, 2018

Sssan520 commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

Sssan520 commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

Sssan520 commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

xuchuanyin commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

CarbonDataQA commented Aug 29, 2018

CarbonDataQA commented Sep 5, 2018

CarbonDataQA commented Jul 24, 2019

CarbonDataQA commented Jul 24, 2019

CarbonDataQA1 commented Feb 3, 2020

Sssan520 commented Aug 24, 2018 •

edited