[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

windpiger · 2017-03-13T07:26:34Z

What changes were proposed in this pull request?

This JIRA is a follow up work after SPARK-19583

As we discussed in that PR

The following DDL for a managed table with an existed default location should throw an exception:

CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
CREATE TABLE ... (PARTITIONED BY ...)

Currently there are some situations which are not consist with above logic:

CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location
situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)
CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
situation: hive table succeed with an existed default location

This PR is going to make above two situations consist with the logic that it should throw an exception
with an existed default location.

How was this patch tested?

unit test added

…e path should throw an exception

windpiger · 2017-03-13T07:27:22Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala

@@ -187,38 +187,32 @@ class InMemoryCatalog(
    val db = tableDefinition.identifier.database.get
    requireDbExists(db)
    val table = tableDefinition.identifier.table
-    if (tableExists(db, table)) {


we have check if the table exists in SessionCatalog.createtable

SparkQA · 2017-03-13T07:28:59Z

Test build #74436 has finished for PR 17272 at commit a0ba419.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T07:29:00Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala

@@ -164,15 +164,6 @@ abstract class ExternalCatalogSuite extends SparkFunSuite with BeforeAndAfterEac
    assert(actual.tableType === CatalogTableType.EXTERNAL)
  }

-  test("create table when the table already exists") {


since in InMemoryCatalog and HiveExternalCatalog does not check if the table exists, here move the test case to SessionCatalogSuite(using InMemoryCatalog) and HiveExternalCatalogSuite

windpiger · 2017-03-13T07:30:28Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala

@@ -155,6 +154,8 @@ case class CreateDataSourceTableAsSelectCommand(
      } else {
        table.storage.locationUri
      }
+
+      sparkSession.sessionState.catalog.checkTableOrPathExists(table, ignoreIfExists = false)


because in following saveDataIntoTable will create the table path and store data in it, we should not check the path in following createTable, we check it here before saveDataIntoTable

SparkQA · 2017-03-13T08:48:40Z

Test build #74437 has finished for PR 17272 at commit 65d7ea9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-13T09:21:08Z

Test build #74441 has finished for PR 17272 at commit 0e753cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T11:31:10Z

retest this please

SparkQA · 2017-03-13T19:03:53Z

Test build #74445 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T23:27:39Z

retest this please

SparkQA · 2017-03-14T01:16:56Z

Test build #74468 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T02:19:07Z

Test build #74472 has finished for PR 17272 at commit cd4a091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T06:17:34Z

Test build #74493 has started for PR 17272 at commit 2ac70b4.

windpiger · 2017-03-14T08:10:02Z

retest this please

SparkQA · 2017-03-14T09:59:52Z

Test build #74499 has finished for PR 17272 at commit 2ac70b4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T12:10:12Z

python/pyspark/sql/tests.py

@@ -1892,7 +1892,7 @@ def test_list_tables(self):
        self.assertEquals(spark.catalog.listTables(), [])
        self.assertEquals(spark.catalog.listTables("some_db"), [])
        spark.createDataFrame([(1, 1)]).createOrReplaceTempView("temp_tab")
-        spark.sql("CREATE TABLE tab1 (name STRING, age INT) USING parquet")
+        spark.sql("CREATE TABLE tbl1 (name STRING, age INT) USING parquet")


python test failed

the location('file:/home/jenkins/workspace/SparkPullRequestBuilder/spark-warehouse/tab1') of table('`default`.`tab1`') already exists.;

the location of tab1 does not deleted , I have tried to found out in which test case it forget to delete it(search all test cases containing tab1 and run it), but they are all ok to delete the location of tab1, so here we change the table name to work around it.

SparkQA · 2017-03-14T13:46:22Z

Test build #74516 has finished for PR 17272 at commit 516c4e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T13:52:52Z

retest this please

SparkQA · 2017-03-14T15:41:36Z

Test build #74524 has finished for PR 17272 at commit 516c4e4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T17:48:02Z

Test build #74535 has finished for PR 17272 at commit 739f207.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-15T14:34:56Z

Test build #74604 has finished for PR 17272 at commit 4351cd7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-05-11T14:46:35Z

@windpiger is this still active?

gatorsmile · 2017-06-19T22:18:30Z

cc @ltnwgl Could you please take this over? Thanks!

gatorsmile · 2018-03-19T19:04:20Z

cc @dongjoon-hyun Are you interested in this PR? Just take it over?

gengliangwang · 2018-03-20T04:01:14Z

@gatorsmile I will take it over :)

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish #17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <gengliang.wang@databricks.com> Closes #20886 from gengliangwang/pr-17272.

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish apache#17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <gengliang.wang@databricks.com> Closes apache#20886 from gengliangwang/pr-17272.

[SPARK-19724][SQL]create a managed table with an existed default tabl…

a0ba419

…e path should throw an exception

windpiger commented Mar 13, 2017

View reviewed changes

windpiger added 2 commits March 13, 2017 15:31

remove empty lines

65d7ea9

fix comment

0e753cc

fix test failed

c8d9b77

fix test failed

cd4a091

windpiger force-pushed the managedTableLocationExists branch from 2ac70b4 to cd4a091 Compare March 14, 2017 12:08

windpiger commented Mar 14, 2017

View reviewed changes

reset catalog in pyspark tests

739f207

windpiger force-pushed the managedTableLocationExists branch from 516c4e4 to 739f207 Compare March 14, 2017 16:00

to find why jenkins failed

4351cd7

HyukjinKwon mentioned this pull request May 17, 2017

[INFRA] Close stale PRs #18017

Closed

asfgit closed this in 5d2750a May 18, 2017

gengliangwang mentioned this pull request Mar 22, 2018

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #20886

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

windpiger commented Mar 13, 2017

windpiger Mar 13, 2017

SparkQA commented Mar 13, 2017

windpiger Mar 13, 2017 •

edited

Loading

windpiger Mar 13, 2017

SparkQA commented Mar 13, 2017

SparkQA commented Mar 13, 2017

windpiger commented Mar 13, 2017

SparkQA commented Mar 13, 2017

windpiger commented Mar 13, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

windpiger commented Mar 14, 2017

SparkQA commented Mar 14, 2017

windpiger Mar 14, 2017

SparkQA commented Mar 14, 2017

windpiger commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 15, 2017

HyukjinKwon commented May 11, 2017

gatorsmile commented Jun 19, 2017

gatorsmile commented Mar 19, 2018

gengliangwang commented Mar 20, 2018

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #17272

Conversation

windpiger commented Mar 13, 2017

What changes were proposed in this pull request?

How was this patch tested?

windpiger Mar 13, 2017

Choose a reason for hiding this comment

SparkQA commented Mar 13, 2017

windpiger Mar 13, 2017 • edited Loading

Choose a reason for hiding this comment

windpiger Mar 13, 2017

Choose a reason for hiding this comment

SparkQA commented Mar 13, 2017

SparkQA commented Mar 13, 2017

windpiger commented Mar 13, 2017

SparkQA commented Mar 13, 2017

windpiger commented Mar 13, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

windpiger commented Mar 14, 2017

SparkQA commented Mar 14, 2017

windpiger Mar 14, 2017

Choose a reason for hiding this comment

SparkQA commented Mar 14, 2017

windpiger commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 15, 2017

HyukjinKwon commented May 11, 2017

gatorsmile commented Jun 19, 2017

gatorsmile commented Mar 19, 2018

gengliangwang commented Mar 20, 2018

windpiger Mar 13, 2017 •

edited

Loading