[SPARK-36841][SQL] Add ansi syntax `set catalog xxx` to change the current catalog #34096

Peng-Lei · 2021-09-24T10:00:15Z

What changes were proposed in this pull request?

1、Add the statement of set catalog xxx to change the current catalog
2、Retain the USE statement to change the current catalog
3、Forcible loading the new catalog when change the new catalog.

Why are the changes needed?

Ansi SQL use SET CATALOG XXX statement to change the catalog.

DISCUSS

Does this PR introduce any user-facing change?

Yes, User can use SET CATALOG XXX to change the current catalog

How was this patch tested?

Add ut testcase

AmplabJenkins · 2021-09-24T10:06:52Z

Can one of the admins verify this patch?

Peng-Lei · 2021-09-24T10:17:24Z

@cloud-fan @imback82 Could you take a look ? Thank you.

Peng-Lei · 2021-09-26T08:09:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala

@@ -120,6 +120,7 @@ class CatalogManager(
  def setCurrentCatalog(catalogName: String): Unit = synchronized {
    // `setCurrentCatalog` is noop if it doesn't switch to a different catalog.
    if (currentCatalog.name() != catalogName) {
+      catalog(catalogName)


Here I just want to check the catalogName is exist and catalog is configed.

cloud-fan · 2021-09-27T08:31:26Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

@@ -107,6 +107,7 @@ statement
    : query                                                            #statementDefault
    | ctes? dmlStatementNoWith                                         #dmlStatement
    | USE NAMESPACE? multipartIdentifier                               #use
+    | SET CATALOG multipartIdentifier                                  #setCatalog


multipartIdentifier? I think it should be identifier | STRING, so that people can write SET CATALOG abc or SET CATALOG '_ 56'

Thank you. I get it.

cloud-fan · 2021-09-27T08:33:10Z

sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala

@@ -2916,6 +2916,22 @@ class DataSourceV2SQLSuite
    }
  }

+  test("SPARK-36481: Test for SET CATALOG statement") {


can we add UT in DDLParserSuite?

imback82 · 2021-09-28T16:25:17Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala

+/**
+ * A SetCatalog statement, as parsed from SQL.
+ */
+case class SetCatalogStatement(nameParts: Option[String]) extends LeafParsedStatement


Can't we just make this nameParts: String without Option?

Thank you for your advice. Done

cloud-fan · 2021-09-29T02:45:57Z

SET CATALOG is a very simple command that we can just implement as RunnableCommand. Can you follow #34128 and simplify this command as well?

Peng-Lei · 2021-09-29T03:35:45Z

SET CATALOG is a very simple command that we can just implement as RunnableCommand. Can you follow #34128 and simplify this command as well?

ok. Thank you for the reminder.

cloud-fan · 2021-09-29T07:44:58Z

sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4

@@ -107,6 +107,7 @@ statement
    : query                                                            #statementDefault
    | ctes? dmlStatementNoWith                                         #dmlStatement
    | USE NAMESPACE? multipartIdentifier                               #use
+    | SET CATALOG catalogIdentifier                                    #setCatalog


is it simpler to do SET CATALOG identifier | STRING

cloud-fan · 2021-09-29T07:48:25Z

sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala

+      } else if (ctx.catalogIdentifier.STRING() != null) {
+        string(ctx.catalogIdentifier().STRING())
+      } else {
+        ctx.catalogIdentifier().getText


We should never hit here. How about throw IllegalStateException here?

OK, How about QueryParsingErrors.invalidCatalogNameError(ctx) with ParseException ?

It's not a user-facing error, so we shouldn't use QueryParsingErrors

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala

Peng-Lei · 2021-09-29T10:45:59Z

@cloud-fan A little off-topic: I also think SHOW CATALOGS #33175 is a very simple command that we can just implement as RunnableCommand follow #34128. Could I continue to update after holiday?

cloud-fan · 2021-10-05T08:12:54Z

Could I continue to update after holiday?

Yes thanks!

cloud-fan · 2021-10-05T09:23:22Z

thanks, merging to master!

github-actions bot added DOCS SQL labels Sep 24, 2021

Peng-Lei commented Sep 26, 2021

View reviewed changes

cloud-fan reviewed Sep 27, 2021

View reviewed changes

Peng-Lei force-pushed the set-catalog-statement branch from 21a9454 to 330a35a Compare September 28, 2021 11:16

imback82 reviewed Sep 28, 2021

View reviewed changes

Peng-Lei force-pushed the set-catalog-statement branch from 330a35a to 3183b4c Compare September 29, 2021 02:06

Peng-Lei requested review from cloud-fan and gengliangwang September 29, 2021 02:10

Peng-Lei force-pushed the set-catalog-statement branch 2 times, most recently from 8d29466 to 7d24ab7 Compare September 29, 2021 07:14

cloud-fan reviewed Sep 29, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala Show resolved Hide resolved

Simplify code

1fbe12a

Peng-Lei force-pushed the set-catalog-statement branch from 7d24ab7 to 1fbe12a Compare September 29, 2021 10:34

cloud-fan approved these changes Oct 5, 2021

View reviewed changes

cloud-fan closed this in 6e8a462 Oct 5, 2021

cloud-fan mentioned this pull request Oct 8, 2021

[SPARK-36849][SQL] Migrate UseStatement to v2 command framework #34127

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-36841][SQL] Add ansi syntax `set catalog xxx` to change the current catalog #34096

[SPARK-36841][SQL] Add ansi syntax `set catalog xxx` to change the current catalog #34096

Peng-Lei commented Sep 24, 2021 •

edited

AmplabJenkins commented Sep 24, 2021

Peng-Lei commented Sep 24, 2021

Peng-Lei Sep 26, 2021

cloud-fan Sep 27, 2021

gengliangwang Sep 27, 2021

Peng-Lei Sep 28, 2021

cloud-fan Sep 27, 2021

Peng-Lei Sep 28, 2021

imback82 Sep 28, 2021

Peng-Lei Sep 29, 2021

cloud-fan commented Sep 29, 2021

Peng-Lei commented Sep 29, 2021

cloud-fan Sep 29, 2021

Peng-Lei Sep 29, 2021

cloud-fan Sep 29, 2021

Peng-Lei Sep 29, 2021

cloud-fan Sep 29, 2021

Peng-Lei commented Sep 29, 2021 •

edited

cloud-fan commented Oct 5, 2021

cloud-fan commented Oct 5, 2021

[SPARK-36841][SQL] Add ansi syntax set catalog xxx to change the current catalog #34096

[SPARK-36841][SQL] Add ansi syntax set catalog xxx to change the current catalog #34096

Conversation

Peng-Lei commented Sep 24, 2021 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

AmplabJenkins commented Sep 24, 2021

Peng-Lei commented Sep 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloud-fan commented Sep 29, 2021

Peng-Lei commented Sep 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Peng-Lei commented Sep 29, 2021 • edited

cloud-fan commented Oct 5, 2021

cloud-fan commented Oct 5, 2021

[SPARK-36841][SQL] Add ansi syntax `set catalog xxx` to change the current catalog #34096

[SPARK-36841][SQL] Add ansi syntax `set catalog xxx` to change the current catalog #34096

Peng-Lei commented Sep 24, 2021 •

edited

Peng-Lei commented Sep 29, 2021 •

edited