Fix MSE explainAskingServers segment grouping and remove all-or-nothing failure#18696
Conversation
…ng failure ExplainNodeSimplifier merged the per-segment children of a combine explain node in an all-or-nothing way: it folded every child into a single plan and, if any pair was not mergeable, hit `assert false` and bailed out keeping all children. This caused two symptoms with explainAskingServers=true: - With assertions disabled (prod default) it returned the full un-grouped plan with one node per segment, producing huge plans (DATA-116), made worse when prefetch wraps each segment in AcquireReleaseColumnsSegment. - With assertions enabled (-ea) the same condition threw AssertionError, which surfaced to the user as a generic INTERNAL "Error while planning query". This is the deterministic, env-dependent explain failure. Replace the all-or-nothing fold with a greedy partial merge (the same approach already used across servers in PlanNodeMerger.mergeCombineChildren): equivalent segment plans collapse into a single group, genuinely different plans are kept as separate groups, and the merge never fails. When more than one distinct plan remains, each group is wrapped in an "Alternative" node annotated with a "segments" count so the reader can see how many segments run each plan. The count uses the default (additive) merge type, so per-server counts are summed as plans merge across servers. Single-plan output is unchanged. Also fix an inverted condition in PlanNodeMerger.visitExchange that reported exchanges over the same tables as not mergeable (and vice versa). Add unit tests for ExplainNodeSimplifier and PlanNodeMerger (none existed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18696 +/- ##
============================================
+ Coverage 64.48% 64.58% +0.09%
Complexity 1291 1291
============================================
Files 3372 3372
Lines 208639 208666 +27
Branches 32590 32606 +16
============================================
+ Hits 134550 134759 +209
+ Misses 63287 63080 -207
- Partials 10802 10827 +25
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
yashmayya
left a comment
There was a problem hiding this comment.
Took a careful pass, mostly on the cross-server merge path since that's where the segment counts get summed.
- The greedy partial merge looks correct. With
verbose=false, "mergeable" is effectively an equivalence relation, so each child lands in exactly one group and the grouping is unambiguous/order-independent. - Confirmed the count summing:
segmentsbeing a DEFAULT/long attribute means the values add when two servers' plans merge, andmergeCombineChildrenpairs theAlternatives by their inner plan (HashSet matching), not by position — so it still holds even when the per-server sort order differs. - The
visitExchangeflip is clearly right; it now lines up with every othervisit*method (!equals-> not mergeable).
Pulled the branch and ran both new test classes — all 12 pass.
One minor, non-blocking thing: when one server has divergent segment plans (so it emits Alternative wrappers) and another server's segments are all uniform (so the simplifier leaves a single bare child), merging them across servers leaves the uniform server's plan as a bare sibling next to the Alternatives, and its segments don't fold into the matching Alternative count. It's explain-only/cosmetic and still much better than the old behavior, so just flagging it as a possible follow-up.
LGTM 👍
Addresses review feedback: previously the per-segment Alternative wrapper was only emitted when a server saw more than one distinct segment plan. When one server had divergent segment plans and another's segments were all uniform, merging across servers left the uniform server's plan as a bare sibling next to the Alternatives, and its segment count did not fold into the matching group. Always wrap each group in an Alternative(segments=N) during per-server simplification, so every server contributes an Alternative that pairs and sums counts across servers. After all servers are merged, removeRedundantAlternatives strips any Alternative that is the sole child of a combine, so combine nodes where all segments share a plan render exactly as before (no test changes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the careful review @yashmayya! Good catch on the uniform-vs-divergent case — pushed a follow-up that addresses it (66e7609). Rather than conditionally emitting the Added unit tests for the cross-server fold ( |
|
Opened the corresponding docs update PR: pinot-contrib/pinot-docs#863 |
…863) ## Summary - explain how multi-stage brief segment plans group distinct per-segment plans into `Alternative(segments=[n])` branches - note that equivalent plans still collapse into one branch with additive stats summed, while the single-plan case still renders without the wrapper ## Source cross-check - `pinot-query-planner/src/main/java/org/apache/pinot/query/planner/explain/ExplainNodeSimplifier.java` - `pinot-query-planner/src/test/java/org/apache/pinot/query/planner/explain/ExplainNodeSimplifierTest.java` ## Validation - `git diff --check`
Problem
With
explainAskingServers=true, the broker asks servers for their physical plans andExplainNodeSimplifiergroups the per-segment children of each combine node. The grouping was all-or-nothing: it folded every child into a single plan and, if any pair was not mergeable, hitassert falseand bailed out keeping every child. This produced two symptoms:AcquireReleaseColumnsSegmentnode.-ea), the same condition threwAssertionError, surfaced to the user as a genericINTERNAL"Error while planning query". Failure vs success was deterministic per environment (does the broker run with-ea?) and per query (do all segment plans merge?).Fix
Replace the all-or-nothing fold with a greedy partial merge — the same approach already used across servers in
PlanNodeMerger.mergeCombineChildren:Alternativenode annotated with asegmentscount, so the reader can tell the plans apart and see how many segments run each one. The count uses the default (additive) merge type, so per-server counts are summed as plans merge across servers (mirrors how alternatives across servers are represented inAskingServerStageExplainer).Example divergent output:
Also fixes an inverted condition in
PlanNodeMerger.visitExchangethat reported exchanges over the same tables as not mergeable (and exchanges over different tables as mergeable).Tests
Adds unit tests for
ExplainNodeSimplifierandPlanNodeMerger(there were none): identical-collapse, summed attributes, partial grouping with counts, the prefetchAcquireReleaseColumnsSegmentshape, no-throw-on-unmergeable, and cross-server count summing. The no-throw test fails against the previous code withAssertionError: Unmergeable inputs found, reproducing the deterministic failure.🤖 Generated with Claude Code