-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-37963][SQL] Need to update Partition URI after renaming table in InMemoryCatalog #35251
Conversation
@@ -74,6 +74,16 @@ trait AlterTableRenameSuiteBase extends command.AlterTableRenameSuiteBase { | |||
} | |||
} | |||
} | |||
|
|||
test("preserve partition info") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we put the test in the base class?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure, updated.
@@ -126,4 +126,14 @@ trait AlterTableRenameSuiteBase extends QueryTest with DDLCommandTestUtils { | |||
spark.sessionState.catalogManager.reset() | |||
} | |||
} | |||
|
|||
test("preserve partition info") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit but maybe we should better add JIRA prefix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, updated
Merging to master/3.2 |
…in InMemoryCatalog ### What changes were proposed in this pull request? After renaming a partitioned table, select from the new table from InMemoryCatalog will get an empty result. The following checkAnswer will fail as the result is empty. ``` sql(s"create table foo(i int, j int) using PARQUET partitioned by (j)") sql("insert into table foo partition(j=2) values (1)") sql(s"alter table foo rename to bar") checkAnswer(spark.table("bar"), Row(1, 2)) ``` To fix the bug, we need to update Partition URI after renaming a table in InMemoryCatalog ### Why are the changes needed? Bug fix ### Does this PR introduce _any_ user-facing change? No, InMemoryCatalog is used internally and HMS doesn't have this bug. ### How was this patch tested? Unit test Closes #35251 from gengliangwang/fixAlterRename. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit 851eb28) Signed-off-by: Gengliang Wang <gengliang@apache.org>
…in InMemoryCatalog ### What changes were proposed in this pull request? After renaming a partitioned table, select from the new table from InMemoryCatalog will get an empty result. The following checkAnswer will fail as the result is empty. ``` sql(s"create table foo(i int, j int) using PARQUET partitioned by (j)") sql("insert into table foo partition(j=2) values (1)") sql(s"alter table foo rename to bar") checkAnswer(spark.table("bar"), Row(1, 2)) ``` To fix the bug, we need to update Partition URI after renaming a table in InMemoryCatalog ### Why are the changes needed? Bug fix ### Does this PR introduce _any_ user-facing change? No, InMemoryCatalog is used internally and HMS doesn't have this bug. ### How was this patch tested? Unit test Closes apache#35251 from gengliangwang/fixAlterRename. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit 851eb28) Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit 4a8e4cb) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
After renaming a partitioned table, select from the new table from InMemoryCatalog will get an empty result.
The following checkAnswer will fail as the result is empty.
To fix the bug, we need to update Partition URI after renaming a table in InMemoryCatalog
Why are the changes needed?
Bug fix
Does this PR introduce any user-facing change?
No, InMemoryCatalog is used internally and HMS doesn't have this bug.
How was this patch tested?
Unit test