[SPARK-37904][SQL] Improve RebalancePartitions in rules of Optimizer#35208
Closed
ulysses-you wants to merge 2 commits intoapache:masterfrom
Closed
[SPARK-37904][SQL] Improve RebalancePartitions in rules of Optimizer#35208ulysses-you wants to merge 2 commits intoapache:masterfrom
ulysses-you wants to merge 2 commits intoapache:masterfrom
Conversation
Contributor
Author
viirya
reviewed
Jan 14, 2022
| val PROJECT: Value = Value | ||
| val RELATION_TIME_TRAVEL: Value = Value | ||
| val REPARTITION_OPERATION: Value = Value | ||
| val REBALANCE: Value = Value |
Contributor
Author
There was a problem hiding this comment.
changed to REBALANCE_PARTITIONS
| // test root node | ||
| val plan1 = emptyRelation.rebalance($"a").analyze | ||
| val optimized1 = Optimize.execute(plan1) | ||
| val expected1 = emptyRelation.analyze |
Member
There was a problem hiding this comment.
nit: expected1 and expected2 are same. Maybe just expected.
Member
|
cc @wangyum too |
viirya
approved these changes
Jan 15, 2022
wangyum
approved these changes
Jan 16, 2022
cloud-fan
approved these changes
Jan 17, 2022
Contributor
|
thanks, merging to master! |
Contributor
Author
|
thank you all ! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Improve
RebalancePartitionsin following rules:NestedColumnAliasingCollapseRepartitionEliminateSortsFoldablePropagationPropagateEmptyRelationBaseWhy are the changes needed?
After SPARK-37267, we support do optimize rebalance partitions in everywhere of plan rather than limit to the root node. So It should make sense to also let
RebalancePartitionswork in all rules of Optimizer likeRepartitionandRepartitionByExpressiondid.Does this PR introduce any user-facing change?
no
How was this patch tested?
Add test in:
NestedColumnAliasingSuiteCollapseRepartitionSuiteEliminateSortsBeforeRepartitionSuiteFoldablePropagationSuitePropagateEmptyRelationSuite