-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21457][SQL] ExternalCatalog.listPartitions should correctly handle partition values with dot #18671
Conversation
// contains dot, and we ask Hive to list partitions w.r.t. the partial partition spec, Hive | ||
// treats dot as matching any single character and may return more partitions than we | ||
// expected. Here we do an extra filter to drop unexpected partitions. | ||
case Some(spec) if spec.exists(_._2.contains(".")) => |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried other special chars but seems dot is the only one having the problem.
And seems Hive does want to do exact matching: https://github.com/apache/hive/blob/release-1.2.1/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L1529-L1535
cc @gatorsmile |
Test build #79709 has finished for PR 18671 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…ndle partition values with dot ## What changes were proposed in this pull request? When we list partitions from hive metastore with a partial partition spec, we are expecting exact matching according to the partition values. However, hive treats dot specially and match any single character for dot. We should do an extra filter to drop unexpected partitions. ## How was this patch tested? new regression test. Author: Wenchen Fan <wenchen@databricks.com> Closes #18671 from cloud-fan/hive. (cherry picked from commit f18b905) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Thanks! Merging to master/2.2 |
…ndle partition values with dot ## What changes were proposed in this pull request? When we list partitions from hive metastore with a partial partition spec, we are expecting exact matching according to the partition values. However, hive treats dot specially and match any single character for dot. We should do an extra filter to drop unexpected partitions. ## How was this patch tested? new regression test. Author: Wenchen Fan <wenchen@databricks.com> Closes apache#18671 from cloud-fan/hive. (cherry picked from commit f18b905) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
What changes were proposed in this pull request?
When we list partitions from hive metastore with a partial partition spec, we are expecting exact matching according to the partition values. However, hive treats dot specially and match any single character for dot. We should do an extra filter to drop unexpected partitions.
How was this patch tested?
new regression test.