[SPARK-57659][CORE] Fix ArrayWrappers int[]/long[] comparator overflow for opposite-sign values#56743
Closed
aviyehuda wants to merge 2 commits into
Closed
[SPARK-57659][CORE] Fix ArrayWrappers int[]/long[] comparator overflow for opposite-sign values#56743aviyehuda wants to merge 2 commits into
aviyehuda wants to merge 2 commits into
Conversation
…w for opposite-sign values
uros-b
reviewed
Jun 24, 2026
uros-b
left a comment
Member
There was a problem hiding this comment.
@aviyehuda Please note that @LuciferYang was already working on this: #56731.
Member
I recommend closing the current PR as a duplicate of existing work. |
Member
|
Closing as duplicate of #56731. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
ComparableIntArray and ComparableLongArray in ArrayWrappers now compare elements with Integer.compare and Long.compare instead of subtraction.
A unit test (testOppositeSignArrayKeyComparison) was added to cover MIN_VALUE vs MAX_VALUE for both int[] and long[].
Why are the changes needed?
ArrayWrappers is used as a key comparator in InMemoryStore for array-typed indices. The previous implementation compared elements via subtraction:
For large opposite-sign values (e.g. Integer.MIN_VALUE vs Integer.MAX_VALUE), subtraction overflows and can return the wrong sign, breaking sort order and map lookups. The long[] path tried to mitigate this with diff > 0 ? 1 : -1, but overflow can still flip the sign of diff, so the result is still wrong.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added testOppositeSignArrayKeyComparison in ArrayWrappersSuite.
build/sbt 'kvstore/testOnly *ArrayWrappersSuite'
Both tests in the suite passed.
Was this patch authored or co-authored using generative AI tooling?
No.