Adopt OrderScheme metadata in cudf-polars#22291
Draft
rjzamora wants to merge 8 commits intorapidsai:mainfrom
Draft
Adopt OrderScheme metadata in cudf-polars#22291rjzamora wants to merge 8 commits intorapidsai:mainfrom
OrderScheme metadata in cudf-polars#22291rjzamora wants to merge 8 commits intorapidsai:mainfrom
Conversation
rjzamora
added
2 - In Progress
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
github-actions
Bot
added
Python
3 tasks
rapids-bot Bot
pushed a commit
that referenced
this pull request
Apr 29, 2026
- Closes #22050 - **Updates `Sort` lowering**: - The "rapidsmpf" runtime no longer uses `Sort(ShuffleSorted(Sort(...)))` - We leave most `Sort` nodes alone. - We map `Sort` directly to `sort_actor` - The `sort_actor` performs it's own top/bottom-k optimization and performs chunk-wise sorting itself. **Motivation**: The `Sort(ShuffleSorted(Sort(...)))` pattern will be too messy when we start utilizing `OrderScheme` metadata (after #22291). The current assumption that the child IR node is always a `Sort` is also not true/safe. **Note**: This PR technically adds code, but it will allow us to *remove* more code once "tasks" is removed (everything `ShuffleSorted` related). Authors: - Richard (Rick) Zamora (https://github.com/rjzamora) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #22315
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NormalizedPartitioningclass #22246OrderSchemepartitioning metadata rapidsmpf#853OrderSchememetadata, strict sort boundaries, and sort-aware execution #22128These changes lay the foundation for tracking and using metadata about the "sortedness" of the data we are passing through the channel in cudf-polars. The next step after this is to attach an
OrderSchemeto theChannelMetadatawithin asort_actor.Checklist