Skip to content

Commit

Permalink
[SPARK-23490][BACKPORT][SQL] Check storage.locationUri with existing …
Browse files Browse the repository at this point in the history
…table in CreateTable

Backport apache#20660 to branch 2.3
=====================================
## What changes were proposed in this pull request?

For CreateTable with Append mode, we should check if `storage.locationUri` is the same with existing table in `PreprocessTableCreation`

In the current code, there is only a simple exception if the `storage.locationUri` is different with existing table:
`org.apache.spark.sql.AnalysisException: Table or view not found:`

which can be improved.

## How was this patch tested?

Unit test

Author: Wang Gengliang <gengliang.wang@databricks.com>

Closes apache#20766 from gengliangwang/backport_20660_to_2.3.
  • Loading branch information
gengliangwang authored and gatorsmile committed Mar 8, 2018
1 parent 86ca915 commit 1dd37ff
Show file tree
Hide file tree
Showing 2 changed files with 37 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,14 @@ case class PreprocessTableCreation(sparkSession: SparkSession) extends Rule[Logi
s"`${existingProvider.getSimpleName}`. It doesn't match the specified format " +
s"`${specifiedProvider.getSimpleName}`.")
}
tableDesc.storage.locationUri match {
case Some(location) if location.getPath != existingTable.location.getPath =>
throw new AnalysisException(
s"The location of the existing table ${tableIdentWithDB.quotedString} is " +
s"`${existingTable.location}`. It doesn't match the specified location " +
s"`${tableDesc.location}`.")
case _ =>
}

if (query.schema.length != existingTable.schema.length) {
throw new AnalysisException(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,35 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}

test("create table - append to a non-partitioned table created with different paths") {
import testImplicits._
withTempDir { dir1 =>
withTempDir { dir2 =>
withTable("path_test") {
Seq(1L -> "a").toDF("v1", "v2")
.write
.mode(SaveMode.Append)
.format("json")
.option("path", dir1.getCanonicalPath)
.saveAsTable("path_test")

val ex = intercept[AnalysisException] {
Seq((3L, "c")).toDF("v1", "v2")
.write
.mode(SaveMode.Append)
.format("json")
.option("path", dir2.getCanonicalPath)
.saveAsTable("path_test")
}.getMessage
assert(ex.contains("The location of the existing table `default`.`path_test`"))

checkAnswer(
spark.table("path_test"), Row(1L, "a") :: Nil)
}
}
}
}

test("Refresh table after changing the data source table partitioning") {
import testImplicits._

Expand Down

0 comments on commit 1dd37ff

Please sign in to comment.