Display input stage id on UnresolvedShuffleExec#1969
Merged
Conversation
Render `stage=N` in UnresolvedShuffleExec's Display output so distributed plans show which input stage each unresolved shuffle reads from. This is a text-only change; the stage_id field already existed and drives serialization, which is unaffected. Update hardcoded plan-string assertions and doc comments in planner.rs and context_checks.rs to match, and regenerate the TPC-H plan-stability goldens (only adds `stage=N,` annotations; stage counts are unchanged).
milenkovicm
approved these changes
Jul 9, 2026
milenkovicm
left a comment
milenkovicm
left a comment
Contributor
There was a problem hiding this comment.
LGTM thanks @andygrove
andygrove
added a commit
to andygrove/datafusion-ballista
that referenced
this pull request
Jul 9, 2026
apache/main now contains apache#1969, the standalone merge of the same UnresolvedShuffleExec stage-id display feature this branch carries in 841d1b2. The display change is identical on both sides; the auto-merge duplicated the new display_includes_stage_id test, so the redundant copy is removed here. Plan-stability goldens regenerated with dev/update-tpch-plan-stability.sh: no golden changes (reuse-aware plans already match the merged code); 26/26 tpch_plan_stability tests pass.
andygrove
added a commit
to andygrove/datafusion-ballista
that referenced
this pull request
Jul 9, 2026
…nge analog) Detect repeated shuffle exchanges during static graph planning and route their consumers to a single shared producer stage, the distributed analog of Spark's ReuseExchange. Identical exchange subtrees (matched by a shared protobuf_canonical_key) collapse to one ShuffleWriter stage instead of being recomputed per consumer, cutting stage count and redundant shuffle work on queries with common sub-plans (e.g. TPC-H q2, q11, q15). - add ballista.optimizer.reuse_exchange_enabled config (default on) - add reuse_shuffle_stages pass and run it in ExecutionGraph planning - extract shared protobuf_canonical_key as the single fingerprint source - make the TPC-H plan-stability suite reuse-aware (fixture applies the reuse pass; goldens q2/q11/q15 regenerated; add exchange_reuse_reduces_stage_count assertion) Rebased onto apache/main; the redundant apache#1967 stage-id display commit is dropped as main now carries it via apache#1969.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1967.
Rationale for this change
When viewing distributed plans,
UnresolvedShuffleExecrenders only its partitioning, so it isn't obvious which input stage it reads from:Including the input stage id makes the stage graph legible directly from the plan text (and from
EXPLAIN/plan-stability snapshots):What changes are included in this PR?
UnresolvedShuffleExec'sDisplay(fmt_as, Default/Verbose) now prefixesstage=Nin both the partitioning and broadcast forms, using the existingstage_idfield. TheTreeRenderform is unchanged.stage_idstruct field, not the Display string, so plan reconstruction is unaffected.planner.rsandclient/tests/context_checks.rs, and regenerated the TPC-H plan-stability goldens (the only change is the addedstage=Nannotation; stage counts are unchanged).Are there any user-facing changes?
Plan text (e.g.
EXPLAIN, executor logs) now shows the input stage id onUnresolvedShuffleExec. No API changes.