Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-35455][SQL] Unify empty relation optimization between normal and AQE optimizer #32602
[SPARK-35455][SQL] Unify empty relation optimization between normal and AQE optimizer #32602
Changes from 21 commits
4514d27
5097247
8df68d9
165077b
2f5fa20
7b80db0
05e074c
48dd92a
8f4dc80
1220087
e26df96
f7a14cf
2c0dfb0
0e151d9
c086f72
9b78ac0
767dd92
47e0c3a
d9ca6da
2fead86
0754936
a6213ea
624e45e
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After more thought, I think this is a big performance issue if we can't propagate empty relations through project/filter which are quite common. The risk of introducing new shuffles is relatively small compared to this.
@ulysses-you can we move all the logic to
PropagateEmptyRelationBase
?PropagateEmptyRelation
should not have any extra logic.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan , if we want to use
transformUpWithPruning
at AQE optimizer side, we need to some more work like add pattern atLogicalQueryStage
. So this PR does not do the change, just usetransformUp
. Do you think it's OK ?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yea it's OK as it's not a regression. cc @sigmod
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@cloud-fan seems we don't need to use
transformUpWithPruning
here since the AQE Optimizer always run once rather than fixed point ?This file was deleted.