Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33650][SQL] Fix the error from ALTER TABLE .. ADD/DROP PARTITION for non-supported partition management table #30594

Closed

Conversation

MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Dec 3, 2020

What changes were proposed in this pull request?

In the PR, I propose to change the order of post-analysis checks for the ALTER TABLE .. ADD/DROP PARTITION command, and perform the general check (does the table support partition management at all) before specific checks.

Why are the changes needed?

The error message for the table which doesn't support partition management can mislead users:

PartitionSpecs are not resolved;;
'AlterTableAddPartition [UnresolvedPartitionSpec(Map(id -> 1),None)], false
+- ResolvedTable org.apache.spark.sql.connector.InMemoryTableCatalog@2fd64b11, ns1.ns2.tbl, org.apache.spark.sql.connector.InMemoryTable@5d3ff859

because it says nothing about the root cause of the issue.

Does this PR introduce any user-facing change?

Yes. After the change, the error message will be:

Table ns1.ns2.tbl can not alter partitions

How was this patch tested?

By running the affected test suite AlterTablePartitionV2SQLSuite.

@github-actions github-actions bot added the SQL label Dec 3, 2020
@SparkQA
Copy link

SparkQA commented Dec 3, 2020

Test build #132147 has finished for PR 30594 at commit b3d1fc9.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @MaxGekk and @cloud-fan .
Merged to master.

@dongjoon-hyun
Copy link
Member

The last commit passed GitHub Action in this PR, but it seems to lead a failure in master branch for some reason.

@MaxGekk
Copy link
Member Author

MaxGekk commented Dec 4, 2020

@dongjoon-hyun Are you sure that this PR causes the failures? Could you point out the error, please.

This https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-3.2-hive-2.3/1753/ succeed.

@dongjoon-hyun
Copy link
Member

@MaxGekk . I'm not blaming here. It's just an incident report.

I also verified the failure locally, but it happens on GitHub Action for now.

@MaxGekk MaxGekk deleted the check-order-AlterTablePartition branch February 19, 2021 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
4 participants