[SPARK-19917][SQL]qualified partition path stored in catalog #17254

windpiger · 2017-03-11T04:43:03Z

What changes were proposed in this pull request?

partition path should be qualified to store in catalog.
There are some scenes:

ALTER TABLE t PARTITION(b=1) SET LOCATION '/path/x'
should be qualified: file:/path/x
Hive 2.0.0 does not support for location without schema here.

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. {0}  is not absolute or has no scheme information.  Please specify a complete absolute uri with scheme information.

ALTER TABLE t PARTITION(b=1) SET LOCATION 'x'
should be qualified: file:/tablelocation/x
Hive 2.0.0 does not support for relative location here.
ALTER TABLE t ADD PARTITION(b=1) LOCATION '/path/x'
should be qualified: file:/path/x
the same with Hive 2.0.0
ALTER TABLE t ADD PARTITION(b=1) LOCATION 'x'
should be qualified: file:/tablelocation/x
the same with Hive 2.0.0

Currently only ALTER TABLE t ADD PARTITION(b=1) LOCATION for hive serde table has the expected qualified path. we should make other scenes to be consist with it.

Another change is for alter table location.

How was this patch tested?

add / modify existing TestCases

windpiger · 2017-03-11T04:47:22Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

@@ -1201,7 +1202,9 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
    verifyLocation(new URI("/swanky/steak/place"))
    // set table partition location without explicitly specifying database
    sql("ALTER TABLE tab1 PARTITION (a='1', b='2') SET LOCATION 'vienna'")
-    verifyLocation(new URI("vienna"), Some(partSpec))
+    val table = spark.sessionState.catalog.getTableMetadata(TableIdentifier("tab1"))


ALTER TABLE PARTITION SET LOCATION
relative location will be quallified with parent path using table location

windpiger · 2017-03-11T04:48:15Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

-      assert(tableLocation.isDefined)
-      makeQualifiedPath(new Path(tableLocation.get.toString, "paris").toString)
-    } else {
-      new URI("paris")


ALTER TABLE ADD PARTITION LOCATION
relative location will be quallified with parent path using table location

now InMemoryCatalog has the same action with HiveExternalCatalog

windpiger · 2017-03-11T04:48:52Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

@@ -2180,6 +2181,13 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {

      withTempDir { dir =>
        assert(!dir.getAbsolutePath.startsWith("file:/"))
+        spark.sql(s"ALTER TABLE t SET LOCATION '$dir'")


ALTER TABLE SET LOCATION
should be also qualified

SparkQA · 2017-03-11T06:17:52Z

Test build #74373 has finished for PR 17254 at commit c0dc3b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-11T06:32:32Z

Test build #74377 has started for PR 17254 at commit 191d8a1.

SparkQA · 2017-03-11T06:37:31Z

Test build #74378 has started for PR 17254 at commit 36a3463.

windpiger · 2017-03-11T08:11:19Z

retest this please

SparkQA · 2017-03-11T10:27:03Z

Test build #74384 has finished for PR 17254 at commit 36a3463.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-11T10:35:09Z

cc @cloud-fan @gatorsmile

SparkQA · 2017-09-06T04:24:28Z

Test build #81437 has finished for PR 17254 at commit 36a3463.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2018-08-24T06:08:13Z

Test build #95195 has started for PR 17254 at commit 36a3463.

remove empty line remove empty line fix test failed fix test failed

SparkQA · 2019-09-17T13:10:54Z

Test build #110779 has finished for PR 17254 at commit f59d65f.

This patch fails R style tests.
This patch does not merge cleanly.
This patch adds no public classes.

windpiger · 2019-09-17T13:18:11Z

retest this please

SparkQA · 2019-09-17T13:28:17Z

Test build #110782 has finished for PR 17254 at commit f61af0a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-17T15:29:05Z

Test build #110783 has finished for PR 17254 at commit c028c75.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-09-17T15:48:16Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+      && !tableDefinition.storage.locationUri.get.isAbsolute) {
+      // make the location of the table qualified.
+      val qualifiedTableLocation =
+      makeQualifiedPath(tableDefinition.storage.locationUri.get)


nit: 2 space indentation

cloud-fan · 2019-09-17T15:55:00Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

@@ -942,7 +954,8 @@ class SessionCatalog(
    requireTableExists(TableIdentifier(table, Option(db)))
    requireExactMatchedPartitionSpec(parts.map(_.spec), getTableMetadata(tableName))
    requireNonEmptyValueInPartitionSpec(parts.map(_.spec))
-    externalCatalog.alterPartitions(db, table, parts)
+


unnecessary blank line

cloud-fan · 2019-09-17T15:56:27Z

Currently only ALTER TABLE t ADD PARTITION(b=1) LOCATION for hive serde table has the expected qualified path.

Where is this done？

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

SparkQA · 2019-09-18T06:07:00Z

Test build #110847 has finished for PR 17254 at commit fadc15c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-09-23T08:01:18Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

+      parts: Seq[CatalogTablePartition]): Seq[CatalogTablePartition] = {
+    parts.map { part =>
+      if (part.storage.locationUri.isDefined && !part.storage.locationUri.get.isAbsolute) {
+        val tbl = getTableMetadata(tableIdentifier)


we should get table metadata only once.

we can mark it as lazy val in case we don't need to qualify any path.

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala

SparkQA · 2019-09-23T12:06:59Z

Test build #111219 has finished for PR 17254 at commit e168b34.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-23T12:15:07Z

Test build #111222 has finished for PR 17254 at commit f8ab46a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-24T05:29:40Z

Test build #111248 has finished for PR 17254 at commit 20e0168.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2019-09-24T06:49:01Z

thanks, merging to master!

windpiger commented Mar 11, 2017

View reviewed changes

dongjoon-hyun added the SQL label Jun 14, 2019

[SPARK-19917][SQL]qualified partition path stored in catalog

f59d65f

remove empty line remove empty line fix test failed fix test failed

windpiger force-pushed the qualifiedPartitionPath branch from 36a3463 to f59d65f Compare September 17, 2019 12:53

windpiger added 2 commits September 17, 2019 21:12

merge with master

99a3c53

fix conflict

f61af0a

fix testcase failed

c028c75

cloud-fan reviewed Sep 17, 2019

View reviewed changes

fix codestyle

fadc15c

cloud-fan reviewed Sep 18, 2019

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala Show resolved Hide resolved

cloud-fan reviewed Sep 23, 2019

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Sep 23, 2019

View reviewed changes

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala Outdated Show resolved Hide resolved

windpiger added 2 commits September 23, 2019 20:03

fix some reveiw issue

e168b34

fix some reveiw issue

f8ab46a

fix style check

20e0168

cloud-fan closed this in da7e5c4 Sep 24, 2019

yaooqinn mentioned this pull request Aug 3, 2020

[SPARK-31709][SQL] Proper base path for database/table location when it is a relative path #28527

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19917][SQL]qualified partition path stored in catalog #17254

[SPARK-19917][SQL]qualified partition path stored in catalog #17254

windpiger commented Mar 11, 2017 •

edited

Loading

windpiger Mar 11, 2017 •

edited

Loading

windpiger Mar 11, 2017 •

edited

Loading

windpiger Mar 11, 2017

SparkQA commented Mar 11, 2017

SparkQA commented Mar 11, 2017

SparkQA commented Mar 11, 2017

windpiger commented Mar 11, 2017

SparkQA commented Mar 11, 2017

windpiger commented Mar 11, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Aug 24, 2018

SparkQA commented Sep 17, 2019

windpiger commented Sep 17, 2019

SparkQA commented Sep 17, 2019

SparkQA commented Sep 17, 2019

cloud-fan Sep 17, 2019

cloud-fan Sep 17, 2019

cloud-fan commented Sep 17, 2019

SparkQA commented Sep 18, 2019

cloud-fan Sep 23, 2019

cloud-fan Sep 23, 2019

SparkQA commented Sep 23, 2019

SparkQA commented Sep 23, 2019

SparkQA commented Sep 24, 2019

cloud-fan commented Sep 24, 2019

[SPARK-19917][SQL]qualified partition path stored in catalog #17254

[SPARK-19917][SQL]qualified partition path stored in catalog #17254

Conversation

windpiger commented Mar 11, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

windpiger Mar 11, 2017 • edited Loading

Choose a reason for hiding this comment

windpiger Mar 11, 2017 • edited Loading

Choose a reason for hiding this comment

windpiger Mar 11, 2017

Choose a reason for hiding this comment

SparkQA commented Mar 11, 2017

SparkQA commented Mar 11, 2017

SparkQA commented Mar 11, 2017

windpiger commented Mar 11, 2017

SparkQA commented Mar 11, 2017

windpiger commented Mar 11, 2017

SparkQA commented Sep 6, 2017

SparkQA commented Aug 24, 2018

SparkQA commented Sep 17, 2019

windpiger commented Sep 17, 2019

SparkQA commented Sep 17, 2019

SparkQA commented Sep 17, 2019

cloud-fan Sep 17, 2019

Choose a reason for hiding this comment

cloud-fan Sep 17, 2019

Choose a reason for hiding this comment

cloud-fan commented Sep 17, 2019

SparkQA commented Sep 18, 2019

cloud-fan Sep 23, 2019

Choose a reason for hiding this comment

cloud-fan Sep 23, 2019

Choose a reason for hiding this comment

SparkQA commented Sep 23, 2019

SparkQA commented Sep 23, 2019

SparkQA commented Sep 24, 2019

cloud-fan commented Sep 24, 2019

windpiger commented Mar 11, 2017 •

edited

Loading

windpiger Mar 11, 2017 •

edited

Loading

windpiger Mar 11, 2017 •

edited

Loading