[SPARK-31113][SQL] Add SHOW VIEWS command #27897

Eric5553 · 2020-03-13T03:01:18Z

What changes were proposed in this pull request?

Previously, user can issue SHOW TABLES to get info of both tables and views.
This PR (SPARK-31113) implements SHOW VIEWS SQL command similar to HIVE to get views only.(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowViews)

Hive -- Only show view names

hive> SHOW VIEWS;
OK
view_1
view_2
...

Spark(Hive-Compatible) -- Only show view names, used in tests and SparkSQLDriver for CLI applications

SHOW VIEWS IN showdb;
view_1
view_2
...

Spark -- Show more information database/viewName/isTemporary

spark-sql> SHOW VIEWS;
userdb	view_1	false
userdb	view_2	false
...

Why are the changes needed?

SHOW VIEWS command provides better granularity to only get information of views.

Does this PR introduce any user-facing change?

Add new SHOW VIEWS SQL command

How was this patch tested?

Add new test show-views.sql and pass existing tests

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala

Eric5553 · 2020-03-14T01:47:28Z

cc @gatorsmile @cloud-fan @maropu , thanks!

maropu · 2020-03-14T06:42:32Z

Could you add a new entry for this command in the SQL doc? See: https://github.com/parano/spark-1/blob/master/docs/sql-ref-syntax-aux-show-tables.md

docs/sql-ref-ansi-compliance.md

maropu · 2020-03-14T06:53:29Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala

@@ -346,6 +346,13 @@ class InMemoryCatalog(
    StringUtils.filterPattern(listTables(db), pattern)
  }

+  override def listViews(db: String, pattern: String): Seq[String] = synchronized {
+    requireDbExists(db)
+    val views = catalog(db).tables.filter(v => v._2.table.tableType == CatalogTableType.VIEW)


nit: .filter(_._2.table.tableType == CatalogTableType.VIEW)

Thanks, updated in c5570d2.

maropu · 2020-03-14T06:56:28Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala

+      parsePlan("SHOW VIEWS IN testcat.ns1.ns2.tbl"),
+      ShowViewsStatement(UnresolvedNamespace(Seq("testcat", "ns1", "ns2", "tbl")), None))
+    comparePlans(
+      parsePlan("SHOW VIEWS IN tbl LIKE '*dog*'"),


Plz add tests for the optional LIKE case? https://github.com/apache/spark/pull/27897/files#diff-8c1cb2af4aa1109e08481dae79052cc3R227

Sure, add/update tests for SHOW VIEWS and SHOW TABLES.

maropu · 2020-03-14T06:57:53Z

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala

-          throw new AnalysisException(
-            s"The database name is not valid: ${ns.quoted}")
+        throw new AnalysisException(
+          s"The database name is not valid: ${ns.quoted}")


nit: (This is not related to this pr though) Can you remove the line break? throw new AnalysisException(s"The database name is not valid: ${ns.quoted}")

Yea, removed the line break for all cases in this file.

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala

sql/core/src/test/resources/sql-tests/inputs/show-views.sql

maropu · 2020-03-14T07:04:36Z

sql/core/src/test/resources/sql-tests/results/show-views.sql.out

+-- !query
+SHOW VIEWS
+-- !query schema
+struct<database:string,viewName:string,isTemporary:boolean>


This output schema is the same with hive? Can you add some running examples (of Spark and Hive) in the PR description?

I just found HiveResult.hiveResultString is used to update query result in Hive compatible form for ShowTablesCommand. Also add transform rule for ShowViewsCommand and updated PR description. Thanks.

Eric5553 · 2020-03-14T15:44:41Z

@maropu Thanks for your review, I'll address them soon!

docs/sql-ref-syntax-aux-show-tables.md

maropu · 2020-03-15T23:22:22Z

docs/sql-ref-syntax-aux-show-views.md

+The `SHOW VIEWS` statement returns all the views for an optionally specified database.
+Additionally, the output of this statement may be filtered by an optional matching
+pattern. If no database is specified then the views are returned from the 
+current database. Note that both global and local temporary views are also returned.


How about Note that both global and local temporary views are also returned. -> Note that the command always lists global and temporary views regardless of a given database?

maropu · 2020-03-15T23:26:32Z

docs/sql-ref-syntax-aux-show-views.md

+SHOW VIEWS [ { FROM | IN } database_name ] [ LIKE 'regex_pattern' ]
+{% endhighlight %}
+
+### Parameters


note: this section is basically the same with the SHOW TABLE one: https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-aux-show-tables.md

docs/sql-ref-syntax-aux-show-views.md

maropu · 2020-03-15T23:34:14Z

docs/sql-ref-syntax-aux-show-views.md

+  | default   | sam        | false        |
+  | default   | sam1       | false        |
+  | default   | suj        | false        |
+  +-----------+------------+--------------+--+


How about adding an example for the isTemporary = true case?

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala

maropu · 2020-03-16T00:30:10Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala

+    tableIdentifierPattern: Option[String]) extends RunnableCommand {
+
+  // The result of SHOW TABLES/SHOW TABLE has three basic columns: database, tableName and
+  // isTemporary. If `isExtended` is true, append column `information` to the output columns.


It seems you wrongly copied&pasted this comment from ShowTablesCommand.

Oops, my mistake. Updated in dc6e3cb.

maropu · 2020-03-16T00:31:48Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala

+
+  override def run(sparkSession: SparkSession): Seq[Row] = {
+    // Since we need to return a Seq of rows, we will call getTables directly
+    // instead of calling tables in sparkSession.


btw, in the original comment in ShowTablesCommand, getTable -> listTables?

maropu · 2020-03-16T00:34:55Z

sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala

+    val db = databaseName.getOrElse(catalog.getCurrentDatabase)
+
+    // Show the information of views.
+    val views =


nit style;

val views = tableIdentifierPattern.map(catalog.listViews(db, _)) .getOrElse(catalog.listViews(db, "*"))

maropu · 2020-03-16T00:39:46Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

@@ -755,6 +755,10 @@ private[hive] class HiveClientImpl(
    client.getTablesByPattern(dbName, pattern).asScala
  }

+  override def listViews(dbName: String, pattern: String): Seq[String] = withHiveState {
+    shim.getTablesByType(client, dbName, pattern, HiveTableType.VIRTUAL_VIEW)
+  }


Plz add tests in the hive side, too.

Yea, updated in dc6e3cb. Not sure if it's enough.

Yea and I think the added ones are good enough.

maropu · 2020-03-17T07:45:34Z

retest this please

maropu · 2020-03-17T13:34:54Z

The overall looks fine now and anyone can check this? @HyukjinKwon @dongjoon-hyun @cloud-fan

docs/sql-ref-syntax-aux-show-views.md

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala

docs/sql-ref-syntax-aux-show-views.md

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

sql/core/src/test/resources/sql-tests/inputs/show-views.sql

cloud-fan · 2020-04-03T07:13:33Z

retest this please

This reverts commit 3073255

cloud-fan · 2020-04-03T11:58:34Z

retest this please

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

SparkQA · 2020-04-06T19:40:29Z

Test build #120878 has finished for PR 27897 at commit 069fbc1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-04-06T20:41:35Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala

@@ -117,6 +117,52 @@ class HiveCommandSuite extends QueryTest with SQLTestUtils with TestHiveSingleto
    }
  }

+  test("show views") {
+    withView("default1a", "default2b", "temp1", "global_temp.temp2") {


Hi, @Eric5553 . Is this correct? Maybe, the following?
default1a -> show1a?
default2b -> show2b?
temp1 -> global_temp.temp1?
global_temp.temp2 -> temp2?

dongjoon-hyun · 2020-04-06T21:33:07Z

Could you add like the following to sql-ref-syntax.md? Currently, it's missing there.

docs/sql-ref-syntax.md
 - [SHOW TBLPROPERTIES](sql-ref-syntax-aux-show-tblproperties.html)
+- [SHOW VIEWS](sql-ref-syntax-aux-show-views.html)
 - [UNCACHE TABLE](sql-ref-syntax-aux-cache-uncache-table.html)

dongjoon-hyun · 2020-04-06T21:55:30Z

docs/sql-ref-syntax-aux-show-views.md

+CREATE VIEW sam1 AS SELECT id, salary FROM employee WHERE name = 'sam1';
+CREATE VIEW suj AS SELECT id, salary FROM employee WHERE name = 'suj';
+USE userdb;
+CREATE VIEW user1 AS SELECT id, salary FROM employee WHERE name = 'user1';


Ur, if employee is a table in default. This will fail. This should be default.employee instead of employee because we switched the database at line 59.

If employee is a temp view, it's valid. However, in this example, we use temp* prefix convention for temp views. So, employee is not a temp view.

dongjoon-hyun · 2020-04-06T22:03:30Z

sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala

@@ -50,6 +50,10 @@ object HiveResult {
    // namespace and table name.
    case command : ShowTablesExec =>
      command.executeCollect().map(_.getString(1))
+    // SHOW VIEWS in Hive only outputs table names while our v1 command outputs
+    // namespace, table name, and isTemp.


table name -> view name?
Maybe, viewName and isTemporary are better.

dongjoon-hyun

Hi, @Eric5553 . I left four comments. For the others, it looks good to me.

withView(".."): [SPARK-31113][SQL] Add SHOW VIEWS command #27897 (comment)
sql-ref-syntax.md: [SPARK-31113][SQL] Add SHOW VIEWS command #27897 (comment)
employee: [SPARK-31113][SQL] Add SHOW VIEWS command #27897 (comment)
viewName: [SPARK-31113][SQL] Add SHOW VIEWS command #27897 (comment)

I also verified this PR with JDK11 with remote HMS as the last testing. Thank you for working on this for a long time.

dongjoon-hyun · 2020-04-06T22:27:20Z

Hi, @juliuszsompolski
After we merge this PR to branch-3.0, is there any other blockers for Simba ODBC Driver?

juliuszsompolski · 2020-04-07T10:44:06Z

@dongjoon-hyun
Simba has been testing with Spark 3.0 and didn't run into other issues.
With SHOW VIEWS the SQLGetTables operation will run without error, but runs into a number of small correctness problems (all of these are just bugs, because it was never the intended behavior to start issuing SHOW VIEWS based on Hive version in Spark driver; it stems from pieces of shared codebase between Spark and Hive).
What current Simba drivers do is collect the answer to SQLGetTables by calling "SHOW SCHEMAS", and then "SHOW TABLES IN schema" and (now with Hive2.3) "SHOW VIEWS IN schema" for every schema. That results in:

local temporary views will be reported in duplicate, once for every schema (Simba did the deduplication of local temp views from SHOW TABLES responses for every schema, but it doesn't apply to SHOW VIEWS as it didn't expect the SHOW VIEWS to happen...)
views in schemas will be reported twice, and both times as "TABLE", not as "VIEW" (once from SHOW TABLES, which currently also returns VIEWS, and once from SHOW VIEWS; because of the way they lookup table types, the occurrence from SHOW VIEWS will also appear as TABLE)
global temporary views will not be reported (it's an existing issue with Spark 2.4 as well; they never look into global_temp schema, because it's not listed by SHOW SCHEMAS).

Simba will fix all these issues in the next release of their ODBC driver, by switching to use the proper SparkGetTablesOperation metadata operation (and other metadata operations implemented by @wangyum and @AngersZhuuuu last year as well).

So with SHOW VIEWS it's also not perfect because it returns duplicate views, but it is better than currently, because it doesn't fail the call and lets PowerBI connect and Tableau show tables/schemas instead of empty in the schema explorer.

Simba audited their behavior that depend on Hive version in the Spark driver and found the following:

If the Hive version is greater than or equal to 0.13.0 the driver expects the following:
- The server supports server side properties.
If the Hive version is greater than or equal to 1.2.0 the driver expects the following:
- The server supports the UNION syntax without requiring the ALL clause before the UNION clause.
If the Hive version is greater than or equal to 2.2.0 the driver expects the following:
- The server supports the SHOW VIEWS syntax.
If the Hive version is greater than or equal to 3.0.0 the driver expects the following:
- The server supports the GetPrimaryKeys and GetCrossReference API.

GetPrimaryKeys and GetCrossReference is a potential future problem, because Spark does not currently override those Hive operations. But Simba will fix it in next release by making SQLPrimaryKeys and SQLForeignKeys just noops - as Spark does not have any use of primary/foreign keys for now.

dongjoon-hyun · 2020-04-07T15:49:31Z

Thank you so much for sharing, @juliuszsompolski !

Eric5553 · 2020-04-07T16:13:23Z

Hi, @dongjoon-hyun, thanks so much for your help! And sorry for my mistakes... I've addressed them as you suggested :-)

dongjoon-hyun · 2020-04-07T16:24:16Z

Thank you for updating, @Eric5553 .
For the last commit, I verified the test case update locally and new generated HTML doc. The other part is only comment-updates. I'll merge this to master/3.0.

### What changes were proposed in this pull request? Previously, user can issue `SHOW TABLES` to get info of both tables and views. This PR (SPARK-31113) implements `SHOW VIEWS` SQL command similar to HIVE to get views only.(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowViews) **Hive** -- Only show view names ``` hive> SHOW VIEWS; OK view_1 view_2 ... ``` **Spark(Hive-Compatible)** -- Only show view names, used in tests and `SparkSQLDriver` for CLI applications ``` SHOW VIEWS IN showdb; view_1 view_2 ... ``` **Spark** -- Show more information database/viewName/isTemporary ``` spark-sql> SHOW VIEWS; userdb view_1 false userdb view_2 false ... ``` ### Why are the changes needed? `SHOW VIEWS` command provides better granularity to only get information of views. ### Does this PR introduce any user-facing change? Add new `SHOW VIEWS` SQL command ### How was this patch tested? Add new test `show-views.sql` and pass existing tests Closes #27897 from Eric5553/ShowViews. Authored-by: Eric Wu <492960551@qq.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit a28ed86) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2020-04-07T16:26:58Z

Thank you again, all!

Eric5553 · 2020-04-07T16:30:09Z

@maropu @cloud-fan @dongjoon-hyun @HyukjinKwon @juliuszsompolski , many thanks!!!

SparkQA · 2020-04-07T21:17:02Z

Test build #120922 has finished for PR 27897 at commit f6132c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

### What changes were proposed in this pull request? Previously, user can issue `SHOW TABLES` to get info of both tables and views. This PR (SPARK-31113) implements `SHOW VIEWS` SQL command similar to HIVE to get views only.(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowViews) **Hive** -- Only show view names ``` hive> SHOW VIEWS; OK view_1 view_2 ... ``` **Spark(Hive-Compatible)** -- Only show view names, used in tests and `SparkSQLDriver` for CLI applications ``` SHOW VIEWS IN showdb; view_1 view_2 ... ``` **Spark** -- Show more information database/viewName/isTemporary ``` spark-sql> SHOW VIEWS; userdb view_1 false userdb view_2 false ... ``` ### Why are the changes needed? `SHOW VIEWS` command provides better granularity to only get information of views. ### Does this PR introduce any user-facing change? Add new `SHOW VIEWS` SQL command ### How was this patch tested? Add new test `show-views.sql` and pass existing tests Closes apache#27897 from Eric5553/ShowViews. Authored-by: Eric Wu <492960551@qq.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Eric5553 commented Mar 13, 2020

View reviewed changes

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 Show resolved Hide resolved

Eric5553 commented Mar 13, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

maropu reviewed Mar 14, 2020

View reviewed changes

docs/sql-ref-ansi-compliance.md Show resolved Hide resolved

maropu reviewed Mar 14, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala Outdated Show resolved Hide resolved

maropu reviewed Mar 14, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/show-views.sql Show resolved Hide resolved

maropu reviewed Mar 14, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

maropu reviewed Mar 16, 2020

View reviewed changes

This comment has been minimized.

Sign in to view

dongjoon-hyun added the SQL label Mar 17, 2020

This comment has been minimized.

Sign in to view

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

docs/sql-ref-syntax-aux-show-views.md Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

docs/sql-ref-syntax-aux-show-views.md Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

docs/sql-ref-syntax-aux-show-views.md Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/show-views.sql Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Mar 17, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/show-views.sql Outdated Show resolved Hide resolved

use toSeq instead of toArray

3073255

This comment has been minimized.

Sign in to view

Revert "use toSeq instead of toArray"

6d68ec4

This reverts commit 3073255

This comment has been minimized.

Sign in to view

cloud-fan reviewed Apr 6, 2020

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala Outdated Show resolved Hide resolved

cloud-fan approved these changes Apr 6, 2020

View reviewed changes

remove unnecessary change

069fbc1

dongjoon-hyun reviewed Apr 6, 2020

View reviewed changes

dongjoon-hyun approved these changes Apr 6, 2020

View reviewed changes

address comments

f6132c0

dongjoon-hyun closed this in a28ed86 Apr 7, 2020

[SPARK-31113][SQL] Add SHOW VIEWS command #27897

[SPARK-31113][SQL] Add SHOW VIEWS command #27897

Conversation

Eric5553 commented Mar 13, 2020 • edited by dongjoon-hyun Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Eric5553 commented Mar 14, 2020

maropu commented Mar 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu Mar 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maropu Mar 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eric5553 commented Mar 14, 2020

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

maropu commented Mar 17, 2020

This comment has been minimized.

maropu commented Mar 17, 2020

This comment has been minimized.

This comment has been minimized.

cloud-fan commented Apr 3, 2020

This comment has been minimized.

This comment has been minimized.

cloud-fan commented Apr 3, 2020

This comment has been minimized.

SparkQA commented Apr 6, 2020

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 6, 2020

Choose a reason for hiding this comment

dongjoon-hyun Apr 6, 2020 • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun left a comment • edited Loading

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 6, 2020

juliuszsompolski commented Apr 7, 2020

dongjoon-hyun commented Apr 7, 2020

Eric5553 commented Apr 7, 2020

dongjoon-hyun commented Apr 7, 2020

dongjoon-hyun commented Apr 7, 2020

Eric5553 commented Apr 7, 2020

SparkQA commented Apr 7, 2020

Eric5553 commented Mar 13, 2020 •

edited by dongjoon-hyun

Loading

maropu commented Mar 14, 2020 •

edited

Loading

maropu Mar 14, 2020 •

edited

Loading

maropu Mar 14, 2020 •

edited

Loading

dongjoon-hyun Apr 6, 2020 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading