Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-33688][SQL] Migrate SHOW TABLE EXTENDED to new resolution framework #30645

Closed
wants to merge 5 commits into from
Closed

[SPARK-33688][SQL] Migrate SHOW TABLE EXTENDED to new resolution framework #30645

wants to merge 5 commits into from

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Dec 7, 2020

What changes were proposed in this pull request?

  1. Remove old statement ShowTableStatement
  2. Introduce new command ShowTableExtended for SHOW TABLE EXTENDED.

This PR is the first step of new V2 implementation of SHOW TABLE EXTENDED, see SPARK-33393.

Why are the changes needed?

This is a part of effort to make the relation lookup behavior consistent: SPARK-29900.

Does this PR introduce any user-facing change?

The changes should not affect V1 tables. For V2, Spark outputs the error:

SHOW TABLE EXTENDED is not supported for v2 tables.

How was this patch tested?

By running SHOW TABLE EXTENDED tests:

$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowTablesSuite"

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 7, 2020

@cloud-fan @imback82 @HyukjinKwon Please, review this PR.

@HyukjinKwon HyukjinKwon mentioned this pull request Dec 7, 2020
Closed
@github-actions github-actions bot added the SQL label Dec 7, 2020
@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36964/

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Test build #132364 has finished for PR 30645 at commit e94f681.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36964/

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 7, 2020

@cloud-fan Don't think the failed test are related to the changes:

org.apache.spark.sql.hive.thriftserver.HiveThriftHttpServerSuite:
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:13884/default?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice;a=avalue;b=bvalue#c=cvalue;d=dvalue: Could not establish connection to jdbc:hive2://localhost:13884/default?hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice;a=avalue;b=bvalue#c=cvalue;d=dvalue: HTTP Response code: 500

@cloud-fan
Copy link
Contributor

retest this please

imback82
imback82 reviewed Dec 7, 2020
View reviewed changes
Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala Outdated
@@ -3199,13 +3199,17 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
}

/**
* Create a [[ShowTableStatement]] command.
* Create a [[ShowTableExtended]] command.
*/
override def visitShowTable(ctx: ShowTableContext): LogicalPlan = withOrigin(ctx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should we update sqlbase.g4 so that this becomes visitShowTableExtended?

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36976/

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36976/

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36983/

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36983/

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Test build #132376 has finished for PR 30645 at commit e94f681.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 7, 2020

Test build #132383 has finished for PR 30645 at commit 4d3c06e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

retest this please

@SparkQA
Copy link

SparkQA commented Dec 8, 2020

Test build #132410 has finished for PR 30645 at commit 4d3c06e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in 2b30dde Dec 8, 2020
@MaxGekk MaxGekk deleted the show-table-extended-statement branch February 19, 2021 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imback82 imback82 imback82 left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

Assignees
No one assigned
Labels
SQL
Projects
None yet
Milestone
No milestone
4 participants
@MaxGekk @SparkQA @cloud-fan @imback82