[SPARK-18961][SQL] Support `SHOW TABLE EXTENDED ... PARTITION` statement #16373

jiangxb1987 · 2016-12-21T08:39:29Z

What changes were proposed in this pull request?

We should support the statement SHOW TABLE EXTENDED LIKE 'table_identifier' PARTITION(partition_spec), just like that HIVE does.
When partition is specified, the SHOW TABLE EXTENDED command should output the information of the partitions instead of the tables.
Note that in this statement, we require exact matched partition spec. For example:

CREATE TABLE show_t1(a String, b Int) PARTITIONED BY (c String, d String);
ALTER TABLE show_t1 ADD PARTITION (c='Us', d=1) PARTITION (c='Us', d=22);

-- Output the extended information of Partition(c='Us', d=1)
SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Us', d=1);
-- Throw an AnalysisException
SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Us');

How was this patch tested?

Add new test sqls in file show-tables.sql.
Add new test case in DDLSuite.

SparkQA · 2016-12-21T10:50:14Z

Test build #70470 has finished for PR 16373 at commit d5ce86a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2016-12-27T03:28:38Z

@yhuai @hvanhovell @gatorsmile Would you please review this PR when you have time? Thanks!

gatorsmile · 2016-12-28T07:58:12Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+        val isTemp = catalog.isTemporaryTable(tableIdent)
+        if (isExtended) {
+          val information = catalog.getTempViewOrPermanentTableMetadata(tableIdent).toString
+          Row(database, tableName, isTemp, s"${information}\n")


Do you know why we need s"${information}\n", instead of information

We have to output a extra '\n' at the end of the whole row, so the output format looks better.

Then, why we do not have the \n in the other similar cases?

I think only in this case, the output could contains multiple rows, and each row contains multiple lines.

This PR also faces the same issue, right? Can you post the outputs of the command?

The statement SHOW TABLE EXTENDED ... PARTITION requires the partition to be fully matched, so in this case, we output no more than one row for each command.
For example:

spark-sql> SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Us', d=1); db2 show_t1 false CatalogPartition( Partition Values: [c=Us, d=1] Storage(Location: file:/Users/meituan/workspace/spark/spark-warehouse/db2.db/show_t1/c=Us/d=1, InputFormat: org.apache.hadoop.mapred.TextInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, Serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, Properties: [serialization.format=1]) Partition Parameters:{transient_lastDdlTime=1482918646}) Time taken: 0.957 seconds, Fetched 1 row(s) spark-sql> SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Us'); Error in query: Partition spec is invalid. The spec (c) must match the partition spec (c, d) defined in table '`db2`.`show_t1`';

BTW, this behavior strictly follows that from HIVE.

Could you post the output of Hive? I am thinking if our outputs look strange. Do we need to improve the function toString?

hive> create table t1(a int, b string) partitioned by (dt string, hour string); hive> alter table t1 add partition (dt='2017-02-08', hour='17'); hive> alter table t1 add partition (dt='2017-02-08', hour='18'); hive> show table extended like 't1' partition(dt='2017-02-08'); FAILED: SemanticException [Error 10006]: Partition not found {dt=2017-02-08} hive> show table extended like 't1' partition(dt='2017-02-08', hour='17'); OK tableName:t1 owner:meituan location:hdfs://localhost:9000/user/hive/warehouse/t1/dt=2017-02-08/hour=17 inputformat:org.apache.hadoop.mapred.TextInputFormat outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat columns:struct columns { i32 a, string b} partitioned:true partitionColumns:struct partition_columns { string dt, string hour} totalNumberFiles:0 totalFileSize:0 maxFileSize:0 minFileSize:0 lastAccessTime:0 lastUpdateTime:1486548513945

gatorsmile · 2016-12-28T08:04:00Z

sql/core/src/test/resources/sql-tests/results/show-tables.sql.out

-== SQL ==
-SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Us')
-^^^
+show_t1


This output is right?

That is weird, I think I should invest a few time on this problem.

The SQLQueryTestSuite runs the sql and convert the result as a hive compatible sequence of strings. It handles the ShowTablesCommand this way:

// SHOW TABLES in Hive only output table names, while ours outputs database, table name, isTemp. case command: ExecutedCommandExec if command.cmd.isInstanceOf[ShowTablesCommand] => command.executeCollect().map(_.getString(1))

We may either change this case, or we create a new command ShowTableCommand to handle the statement SHOW TABLE EXTENDED LIKE ... [PARTITION].

hiveResultString is only used for testing. I think we should fix it as long as it does not break any test case.

SparkQA · 2017-02-08T10:31:59Z

Test build #72575 has finished for PR 16373 at commit 0020c16.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-08T13:18:34Z

Test build #72579 has finished for PR 16373 at commit 0424edd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-07T10:52:10Z

Test build #74090 has finished for PR 16373 at commit 1b2a41c.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-07T12:52:53Z

Test build #74091 has finished for PR 16373 at commit fa93c80.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-03-08T01:59:44Z

@gatorsmile Does this PR looks good?

jiangxb1987 · 2017-03-10T23:54:21Z

@cloud-fan @hvanhovell @gatorsmile Would you please look at this PR?

gatorsmile · 2017-03-11T19:25:26Z

Will review this today. :)

gatorsmile · 2017-03-12T15:49:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

@@ -123,8 +123,12 @@ class QueryExecution(val sparkSession: SparkSession, val logical: LogicalPlan) {
            .mkString("\t")
      }
    // SHOW TABLES in Hive only output table names, while ours outputs database, table name, isTemp.
-    case command: ExecutedCommandExec if command.cmd.isInstanceOf[ShowTablesCommand] =>
-      command.executeCollect().map(_.getString(1))
+    case command@ ExecutedCommandExec(showTables: ShowTablesCommand) =>


Nit: command@ -> command @

gatorsmile · 2017-03-12T15:53:11Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

-      command.executeCollect().map(_.getString(1))
+    case command@ ExecutedCommandExec(showTables: ShowTablesCommand) =>
+      if (showTables.isExtended) {
+        command.executeCollect().map(_.getString(3))


If we do not have it, any test case failed?

This function is just for testing. If we do not have the related Hive output comparison. We can simplify it to

case command @ ExecutedCommandExec(showTables: ShowTablesCommand) if !showTables.isExtended => command.executeCollect().map(_.getString(1))

gatorsmile · 2017-03-12T15:59:09Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+        val isTemp = catalog.isTemporaryTable(tableIdent)
+        if (isExtended) {
+          val information = catalog.getTempViewOrPermanentTableMetadata(tableIdent).toString
+          Row(database, tableName, isTemp, s"${information}\n")


Nit: -> s"$information\n"

gatorsmile · 2017-03-12T16:00:30Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+      //
+      // Note: tableIdentifierPattern should be non-empty, otherwise a [[ParseException]]
+      // should have been thrown by the sql parser.
+      val tableIdentifier = TableIdentifier(tableIdentifierPattern.get, Some(db))


To be consistent, we can rename it to tableIdent

gatorsmile · 2017-03-12T16:02:14Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+      val tableName = table.table
+      val isTemp = catalog.isTemporaryTable(table)
+      val information = partition.toString
+      Seq(Row(database, tableName, isTemp, information))


What is the reason this is different from the above case? That is, why not adding \n?

gatorsmile · 2017-03-12T16:04:18Z

What is the reason why we only have the negative one in the newly added test cases?

cloud-fan · 2017-03-13T04:00:41Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+      val tables =
+        tableIdentifierPattern.map(catalog.listTables(db, _)).getOrElse(catalog.listTables(db))
+      tables.map { tableIdent =>
+        val database = tableIdent.database.getOrElse("")


shall we get the current database?

For temporary views, they have empty database.

cloud-fan · 2017-03-13T04:03:18Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

      }
+    } else {
+      // Show the information of partitions.


when users specify the partition spec, we can only list one table?

Yes - this follows the behavior that Hive does.

jiangxb1987 · 2017-03-13T15:33:51Z

@gatorsmile Added the positive test cases.

SparkQA · 2017-03-13T18:24:02Z

Test build #74451 has finished for PR 16373 at commit 77ca941.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-13T19:05:02Z

sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala

-    // SHOW TABLES in Hive only output table names, while ours outputs database, table name, isTemp.
-    case command: ExecutedCommandExec if command.cmd.isInstanceOf[ShowTablesCommand] =>
+    // SHOW TABLES in Hive only output table names, while ours output database, table name, isTemp.
+    case command @ ExecutedCommandExec(showTables: ShowTablesCommand) =>


case command @ ExecutedCommandExec(showTables: ShowTablesCommand) ->
case command @ ExecutedCommandExec(_: ShowTablesCommand)

gatorsmile · 2017-03-13T19:05:41Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

+      val tables =
+        tableIdentifierPattern.map(catalog.listTables(db, _)).getOrElse(catalog.listTables(db))
+      tables.map { tableIdent =>
+        val database = tableIdent.database.getOrElse(db)


For temporary views, database is empty. val database = tableIdent.database.getOrElse("")

gatorsmile · 2017-03-13T19:11:19Z

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala

@@ -925,6 +925,26 @@ class DDLSuite extends QueryTest with SharedSQLContext with BeforeAndAfterEach {
    }
  }

+  test("show table extended ... partition") {


Just a q. What is the reason why we are unable to do it in show-tables.sql?

In show-tables.sql, we only output the value of the column tableName, we should verify the schema here.

If we change hiveResultString to

case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended => command.executeCollect().map(_.getString(1))

I did a try. It works. Below is the output.

-- !query 22 SHOW TABLE EXTENDED LIKE 'show_t1' PARTITION(c='Ch', d=1) -- !query 22 schema struct<database:string,tableName:string,isTemporary:boolean,information:string> -- !query 22 output showdb show_t1 false CatalogPartition( Partition Values: [c=Ch, d=1] Storage(Location: file:/Users/xiao/IdeaProjects/sparkDelivery/sql/core/spark-warehouse/showdb.db/show_t1/c=Ch/d=1) Partition Parameters:{})

Yes it works, but it outputs the absolute path for Location, so the test suite will fail on another environment.

Then, you just need to improve the function getNormalizedResult in SQLQueryTestSuite to mask it.

Okay, I'll update that later.

SparkQA · 2017-03-14T05:20:39Z

Test build #74481 has finished for PR 16373 at commit b46d771.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…time.

SparkQA · 2017-03-14T12:36:48Z

Test build #74510 has finished for PR 16373 at commit 62aee59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T12:46:12Z

Test build #74506 has finished for PR 16373 at commit bede7ae.

This patch fails PySpark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-03-14T13:28:29Z

retest this please.

SparkQA · 2017-03-14T15:35:17Z

Test build #74520 has finished for PR 16373 at commit 62aee59.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-14T17:06:16Z

LGTM

gatorsmile · 2017-03-14T17:14:14Z

Thanks! Merging to master.

…ABLE EXTENDED ### What changes were proposed in this pull request? Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up #16373 and #15515. ### Why are the changes needed? To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes #30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…ABLE EXTENDED ### What changes were proposed in this pull request? Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up #16373 and #15515. ### Why are the changes needed? To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes #30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 29096a8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…ABLE EXTENDED Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up apache#16373 and apache#15515. To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. Yes. By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Closes apache#30618 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit 29096a8) Signed-off-by: Max Gekk <max.gekk@gmail.com>

…HOW TABLE EXTENDED ### What changes were proposed in this pull request? Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up #16373 and #15515. ### Why are the changes needed? To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *HiveCatalogedDDLSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Authored-by: Max Gekk <max.gekkgmail.com> Signed-off-by: HyukjinKwon <gurwls223apache.org> (cherry picked from commit 29096a8) Signed-off-by: Max Gekk <max.gekkgmail.com> Closes #30640 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive-3.0. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

…HOW TABLE EXTENDED ### What changes were proposed in this pull request? Invoke the check `DDLUtils.verifyPartitionProviderIsHive()` from V1 implementation of `SHOW TABLE EXTENDED` when partition specs are specified. This PR is some kind of follow up #16373 and #15515. ### Why are the changes needed? To output an user friendly error with recommendation like **" ... partition metadata is not stored in the Hive metastore. To import this information into the metastore, run `msck repair table tableName` "** instead of silently output an empty result. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the affected test suites, in particular: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *HiveCatalogedDDLSuite" $ build/sbt -Phive-2.3 -Phive-thriftserver "hive/test:testOnly *PartitionProviderCompatibilitySuite" ``` Authored-by: Max Gekk <max.gekkgmail.com> Signed-off-by: HyukjinKwon <gurwls223apache.org> (cherry picked from commit 29096a8) Signed-off-by: Max Gekk <max.gekkgmail.com> Closes #30641 from MaxGekk/show-table-extended-verifyPartitionProviderIsHive-2.4. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

gatorsmile reviewed Dec 28, 2016

View reviewed changes

jiangxb1987 force-pushed the show-partition-extended branch from d5ce86a to 0020c16 Compare February 8, 2017 08:42

jiangxb1987 force-pushed the show-partition-extended branch from 0424edd to 1b2a41c Compare March 7, 2017 10:45

gatorsmile reviewed Mar 12, 2017

View reviewed changes

cloud-fan reviewed Mar 13, 2017

View reviewed changes

jiangxb1987 force-pushed the show-partition-extended branch from fa93c80 to 77ca941 Compare March 13, 2017 15:15

gatorsmile reviewed Mar 13, 2017

View reviewed changes

jiangxb1987 added 2 commits March 14, 2017 18:31

support SHOW TABLE EXTENDED ... PARTITION statement.

e275d26

update hiveResultString

aa62dc8

jiangxb1987 added 5 commits March 14, 2017 18:31

bugfix

b1b40f4

bugfix

4561fde

refactor.

bc590b7

bugfix

f42ce23

improve function getNormalizedResult() to mask absolute path and date…

62aee59

…time.

jiangxb1987 force-pushed the show-partition-extended branch from bede7ae to 62aee59 Compare March 14, 2017 10:36

asfgit closed this in a02a0b1 Mar 14, 2017

jiangxb1987 deleted the show-partition-extended branch March 16, 2017 06:42

cloud-fan mentioned this pull request Oct 17, 2019

[SPARK-29092][SQL] Report additional information about DataSourceScanExec in EXPLAIN FORMATTED #26042

Closed

MaxGekk mentioned this pull request Dec 5, 2020

[SPARK-33670][SQL] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED #30618

Closed

MaxGekk mentioned this pull request Dec 7, 2020

[SPARK-33670][SQL][3.0] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED #30640

Closed

MaxGekk mentioned this pull request Dec 7, 2020

[SPARK-33670][SQL][2.4] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED #30641

Closed

[SPARK-18961][SQL] Support SHOW TABLE EXTENDED ... PARTITION statement #16373

[SPARK-18961][SQL] Support SHOW TABLE EXTENDED ... PARTITION statement #16373

Conversation

jiangxb1987 commented Dec 21, 2016 • edited

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Dec 21, 2016

jiangxb1987 commented Dec 27, 2016

Choose a reason for hiding this comment

jiangxb1987 Dec 28, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangxb1987 Feb 8, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 8, 2017

SparkQA commented Feb 8, 2017

SparkQA commented Mar 7, 2017

SparkQA commented Mar 7, 2017

jiangxb1987 commented Mar 8, 2017

jiangxb1987 commented Mar 10, 2017 • edited

gatorsmile commented Mar 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gatorsmile commented Mar 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiangxb1987 commented Mar 13, 2017

SparkQA commented Mar 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

SparkQA commented Mar 14, 2017

jiangxb1987 commented Mar 14, 2017

SparkQA commented Mar 14, 2017

gatorsmile commented Mar 14, 2017

gatorsmile commented Mar 14, 2017

[SPARK-18961][SQL] Support `SHOW TABLE EXTENDED ... PARTITION` statement #16373

[SPARK-18961][SQL] Support `SHOW TABLE EXTENDED ... PARTITION` statement #16373

jiangxb1987 commented Dec 21, 2016 •

edited

jiangxb1987 Dec 28, 2016 •

edited

jiangxb1987 Feb 8, 2017 •

edited

jiangxb1987 commented Mar 10, 2017 •

edited