You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Proposal: Shuffle backend interface for Daft extensions
#7462
#4791 requested Celeborn integration for distributed push-based shuffle. That work produced a concrete in-tree implementation in #7453.
Maintainers have said they do not want additional core shuffle backends beyond Flight and Ray, and would rather support this kind of integration as an out-of-tree extension (daft-celeborn or similar) behind a stable shuffle interface.
This discussion is the place to design that interface. The Celeborn issue and PR remain the motivating use case; they should not stay the design thread.
Related work
Issue
#4791 — Integrate with Celeborn to support distributed push-based shuffle
PRs
#7453 — Draft: add Celeborn shuffle backend (in-tree). The integration that raised the extension-interface question.
#6456 — Generic shuffle interface for Flight shuffle (merged). Plan-layer ShuffleWrite / ShuffleRead and backend-parameterized local plan nodes.
Today ShuffleBackend is a closed enum: Ray | Flight { ... }. Native extensions (daft-ext) can register scalar and aggregate functions, not shuffle backends. #6456 genericized the plan and task layer so Flight is no longer a special-case input path, but adding a third backend still means new enum variants and in-tree execution code.
What a shuffle extension likely needs
From the Celeborn integration and the Flight/Ray backends, the shared lifecycle looks like:
Backend initialization (per application / per worker)
Shuffle registration
Map-side write + completion
Reduce-side partition read
Cleanup
The map/reduce handoff is not the same for every backend:
Ray passes object references
Flight passes refs to partitions hosted by Daft workers
Celeborn stores data on Celeborn workers; reducers fetch by logical partition without carrying map-output refs between stages
So the extension boundary is probably larger than “swap the reader and writer.” One direction from the #4791 thread: Daft keeps scheduling and common task metadata; the extension owns initialization, storage, transport, and cleanup; an opaque, serializable shuffle handle carries backend-specific state through planning, task serialization, and worker execution.
Native ABI extensions today do not cover this. A shuffle backend extension may need a different (or additional) hook: Rust-with-Python-bindings, pure Python, or a new ABI surface.
Questions
Boundary. Is the interface readers/writers plus partition refs, or the full shuffle lifecycle (init, register, write, complete, read, cleanup)?
Handle. Is an opaque serializable shuffle handle the right way to avoid a new in-tree enum variant per extension?
Registration. How does an extension register a backend and select it per runner (shuffle_algorithm, session config, or something else)?
Host vs extension. What must stay in Daft (scheduling, task identity, retry metadata, lineage) vs what the extension owns?
SDK. Should this live on the existing daft-ext / native ABI path, a Rust trait in Daft with out-of-tree crates, or a Python plugin hook?
In-tree backends. Do Ray and Flight stay first-class enum variants, with the extension path only for third-party backends?
Celeborn next step. If this interface is the right direction, should #7453 stay as a design reference and move implementation to a separate daft-celeborn package rather than land in Daft core?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Why this discussion
#4791 requested Celeborn integration for distributed push-based shuffle. That work produced a concrete in-tree implementation in #7453.
Maintainers have said they do not want additional core shuffle backends beyond Flight and Ray, and would rather support this kind of integration as an out-of-tree extension (
daft-celebornor similar) behind a stable shuffle interface.This discussion is the place to design that interface. The Celeborn issue and PR remain the motivating use case; they should not stay the design thread.
Related work
Issue
PRs
ShuffleWrite/ShuffleReadand backend-parameterized local plan nodes.Adjacent discussions / roadmap
Current shape in Daft
Today
ShuffleBackendis a closed enum:Ray|Flight { ... }. Native extensions (daft-ext) can register scalar and aggregate functions, not shuffle backends. #6456 genericized the plan and task layer so Flight is no longer a special-case input path, but adding a third backend still means new enum variants and in-tree execution code.What a shuffle extension likely needs
From the Celeborn integration and the Flight/Ray backends, the shared lifecycle looks like:
The map/reduce handoff is not the same for every backend:
So the extension boundary is probably larger than “swap the reader and writer.” One direction from the #4791 thread: Daft keeps scheduling and common task metadata; the extension owns initialization, storage, transport, and cleanup; an opaque, serializable shuffle handle carries backend-specific state through planning, task serialization, and worker execution.
Native ABI extensions today do not cover this. A shuffle backend extension may need a different (or additional) hook: Rust-with-Python-bindings, pure Python, or a new ABI surface.
Questions
shuffle_algorithm, session config, or something else)?daft-ext/ native ABI path, a Rust trait in Daft with out-of-tree crates, or a Python plugin hook?daft-celebornpackage rather than land in Daft core?cc @srilman @yugan95 @jaychia
All reactions