Bug-16563: Imposed an ordering on the filter expressions while checking for equality and hashCode of spark scans.#16570
Open
ahshahid wants to merge 4 commits into
Open
Bug-16563: Imposed an ordering on the filter expressions while checking for equality and hashCode of spark scans.#16570ahshahid wants to merge 4 commits into
ahshahid wants to merge 4 commits into
Conversation
…ions while checking for equality and hashCode so that structurally same scans do not mismatch. This is critical for re-use of exchange to happen , where the pushdown of filters may differ in order, due to spark code using Set type collection for collecting filters for pushdown, resulting in unpredictable ordering of filters when pushed
ahshahid
changed the title
Author
|
Once the tests suite completes, will remove the WIP |
ahshahid
changed the title
…ing reordering causing assertion failures. Instead of doing full string matches, matching individual filter string
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Impose order on filter expressions ( runtime as well as data) while checking for equality and hashCode of Spark Scans, so that structurally same scans do not mismatch. This is critical for re-use of exchange to happen , where the pushdown of filters may differ in order, due to spark code using Set type collection for collecting filters for pushdown, resulting in unpredictable ordering of filters when pushed.
The change involves flattening the expressions and then ordering them based, on each Predicate's string representation.
Once this code patch is reviewed, same modifications may be done in other spark versions too, if required backport for older versions.