New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-35437][SQL][FOLLOWUP] Relax cast if does not need timezone with PrunePartitionsFastFallback #34981
Conversation
Kubernetes integration test starting |
Test build #146470 has finished for PR 34981 at commit
|
Kubernetes integration test status failure |
Can you link to the original PR in your PR description? |
updated |
thanks, merging to master! |
thank you @cloud-fan |
…h PrunePartitionsFastFallback Allow `Cast` during `prunePartitionsFastFallback` if it actually does not require the timezone. The original PR apache#34431 Make `prunePartitionsFastFallback` work with more cases. For example, one partition column data type is `StringType` but the query with the partition predicate using `IntegerType` ```sql CREATE TABLE t(c int) PARTITIONED BY(p string); SELECT * FROM t WHERE p = 20210101; ``` no, not released and this pr itself just affects the perf. Add some test for `Cast` who need or does not need time zone. Closes apache#34981 from ulysses-you/SPARK-35437-FOLLOWUP. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit d1cf8ff)
…h PrunePartitionsFastFallback ### What changes were proposed in this pull request? Allow `Cast` during `prunePartitionsFastFallback` if it actually does not require the timezone. The original PR apache#34431 ### Why are the changes needed? Make `prunePartitionsFastFallback` work with more cases. For example, one partition column data type is `StringType` but the query with the partition predicate using `IntegerType` ```sql CREATE TABLE t(c int) PARTITIONED BY(p string); SELECT * FROM t WHERE p = 20210101; ``` ### Does this PR introduce _any_ user-facing change? no, not released and this pr itself just affects the perf. ### How was this patch tested? Add some test for `Cast` who need or does not need time zone. Closes apache#34981 from ulysses-you/SPARK-35437-FOLLOWUP. Authored-by: ulysses-you <ulyssesyou18@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
Allow
Cast
duringprunePartitionsFastFallback
if it actually does not require the timezone.The original PR #34431
Why are the changes needed?
Make
prunePartitionsFastFallback
work with more cases.For example, one partition column data type is
StringType
but the query with the partition predicate usingIntegerType
Does this PR introduce any user-facing change?
no, not released and this pr itself just affects the perf.
How was this patch tested?
Add some test for
Cast
who need or does not need time zone.