Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-34576][SQL][3.1] Fix drop/add columns to a dataset of DESCRIBE COLUMN #31727

Closed
wants to merge 1 commit into from

Conversation

AngersZhuuuu
Copy link
Contributor

What changes were proposed in this pull request?

In the PR, I propose to generate "stable" output attributes per the logical node of the DESCRIBE COLUMN command.

Why are the changes needed?

This fixes the issue demonstrated by the example:

val tbl = "testcat.ns1.ns2.tbl"
sql(s"CREATE TABLE $tbl (c0 INT) USING _")
val description = sql(s"DESCRIBE TABLE $tbl c0")
description.drop("info_name")
[info]   org.apache.spark.sql.AnalysisException: Resolved attribute(s) info_name#74 missing from info_name#25,info_value#26 in operator !Project [info_name#74]. Attribute(s) with the same name appear in the operation: info_name. Please check if the right attribute(s) are used.;
[info] !Project [info_name#74]
[info] +- LocalRelation [info_name#25, info_value#26]

Does this PR introduce any user-facing change?

After this change user drop()/add() works well.

How was this patch tested?

Added UT

@AngersZhuuuu
Copy link
Contributor Author

FYI @cloud-fan

@SparkQA
Copy link

SparkQA commented Mar 3, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40296/

@SparkQA
Copy link

SparkQA commented Mar 3, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40296/

@SparkQA
Copy link

SparkQA commented Mar 3, 2021

Test build #135714 has finished for PR 31727 at commit 6e79c5c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@github-actions github-actions bot added the SQL label Mar 3, 2021
@AngersZhuuuu
Copy link
Contributor Author

retest this please

@SparkQA
Copy link

SparkQA commented Mar 4, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40310/

@SparkQA
Copy link

SparkQA commented Mar 4, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40310/

@SparkQA
Copy link

SparkQA commented Mar 4, 2021

Test build #135728 has finished for PR 31727 at commit 6e79c5c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

seems like a real test failure.

@AngersZhuuuu
Copy link
Contributor Author

@cloud-fan Seems we should backport more pr to 3.1 branch according to the failed test.

@cloud-fan
Copy link
Contributor

ah we don't have v2 DESC COLUMN in 3.1, so the bug doesn't exist there. Let's close this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
3 participants