[SPARK-45925][SQL] Making SubqueryBroadcastExec equivalent to SubqueryAdaptiveBroadcastExec #43807
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
November 13, 2023 21:36
… by also canonicalizing the build plan. Also making SubqueryBroadcastExec equivalent to SubqueryAdaptiveBroadcastExec. Similar equivalence will also be done for SubquderyBroadcastExec under separate PR
…ptiveBroadcastExec
ahshahid
changed the title
HyukjinKwon
changed the title
|
cc @ulysses-you and @peter-toth FYI |
|
The other option is that we make the canonicalized form of both SubqueryAdaptiveBroadcastExec and SubqueryBroadcastExec to be of type SubqueryBroadcastExec. that way equals and hashCode will go away and it will serve our purpose. |
|
I think if I get clean run on PR SPARK-45924, then this PR can be closed without merging |
|
@beliefer I think you may be right. In my another PR for broadcast-var-pushdown, I am seeing unmodified SubqueryAdaptiveBroadcastExec in the stage cache 's keys. May be it is an issue in my code or something else. Will check my code again for this. So as of now, I think it makes sense to close this PR and also the other PR in SubqueryBroadcastHashExec |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Implementing equals and hashCode in SubqueryBroadcastExec so that it is made equivalent to SubqueryAdaptiveBroadcastExec . During the bug testing and fix for re-use of exchange in AQE, it turned out that the first exchange got created with SubqueryAdaptiveBroadcastExec, while another incoming exchange which is as such identical to the existing one, but differed as it was having SubqueryBroadcastExec, which contributed to re-use of exchange not happening.
Why are the changes needed?
Needed so that reuse of exchange happens correctly in AQE
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added test for checking the equivalence with SubqueryAdaptiveBroadcastExec.
Was this patch authored or co-authored using generative AI tooling?
No