[SPARK-58587][ML] Move final IsotonicRegression PAV pass to executor - #57788
Closed
zhengruifeng wants to merge 2 commits into
Closed
[SPARK-58587][ML] Move final IsotonicRegression PAV pass to executor#57788zhengruifeng wants to merge 2 commits into
zhengruifeng wants to merge 2 commits into
Conversation
zhengruifeng
changed the title
zhengruifeng
changed the title
zhengruifeng
marked this pull request as ready for review
August 5, 2026 08:56
zhengruifeng
changed the title
uros-b
approved these changes
Aug 5, 2026
Member
|
LGTM |
zhengruifeng
added a commit
that referenced
this pull request
Aug 6, 2026
### What changes were proposed in this pull request? This PR moves the final Pool Adjacent Violators (PAV) pass in `IsotonicRegression` from the driver to a single executor partition. It replaces the driver-side in-memory sort of all partial results with `sortBy(..., numPartitions = 1)`, allowing Spark shuffle sort to spill while ordering partial results. The driver collects only the final PAV output. The PR also adds regression coverage for a monotonicity violation that spans two input partitions. ### Why are the changes needed? The final PAV pass can substantially compress partial results. Running it before `collect()` reduces driver memory and data transfer when that compression occurs. The final PAV pass remains single-partition, but the preceding sort no longer materializes every partial result on the driver. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a regression test for merging partial PAV results across partitions. Static checks completed successfully: - `git diff --check` - ASCII scan of the changed Scala files - changed-line length scan The test suite was not run because it was not requested. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) Closes #57788 from zhengruifeng/ml_iso_fit. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com> (cherry picked from commit 80afd1d) Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR moves the final Pool Adjacent Violators (PAV) pass in
IsotonicRegressionfrom thedriver to a single executor partition. It replaces the driver-side in-memory sort of all partial
results with
sortBy(..., numPartitions = 1), allowing Spark shuffle sort to spill while orderingpartial results. The driver collects only the final PAV output.
The PR also adds regression coverage for a monotonicity violation that spans two input
partitions.
Why are the changes needed?
The final PAV pass can substantially compress partial results. Running it before
collect()reduces driver memory and data transfer when that compression occurs. The final PAV pass remains
single-partition, but the preceding sort no longer materializes every partial result on the driver.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added a regression test for merging partial PAV results across partitions.
Static checks completed successfully:
git diff --checkThe test suite was not run because it was not requested.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)