[fix](local shuffle) Require hash input for distinct finalize agg without group keys - #66570
Open
924060929 wants to merge 1 commit into
Open
[fix](local shuffle) Require hash input for distinct finalize agg without group keys#66570924060929 wants to merge 1 commit into
924060929 wants to merge 1 commit into
Conversation
…roup keys The FE local-shuffle planner handed a NoRequire distribution to a finalize merge agg that has no group keys but DISTINCT aggregates (e.g. count(distinct k)). Unlike COUNT(*), such an agg emits per-instance scalar values that the parent sums (sum0(multi_distinct_count(...))), so its input must be hash-partitioned by the distinct key. When a PASSTHROUGH local exchange (e.g. broadcast-join probe fan-out) scatters same-key rows across instances, the parent double-counts overlapping keys — result = correct value × local task count. Mirror BE AggSinkOperatorX::update_operator's _partition_exprs (grouping exprs, or distinct/distribute exprs): aggs with a partition requirement must demand HASH from their child; only partition-less aggs (COUNT(*)-style) keep NoRequire. requiresShuffleForCorrectness now also covers DISTINCT aggregates to match BE is_shuffled_operator(). Regression tests: AggregationNode unit coverage for every phase/flag combination, plus a sql-level distributed-plan test asserting the LOCAL_HASH exchange appears below the distinct finalize agg.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
morrySnow
changed the title
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29178 ms |
Contributor
TPC-DS: Total hot run time: 157704 ms |
Contributor
ClickBench: Total hot run time: 23.68 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With the FE local-shuffle planner enabled (default
enable_local_shuffle_planner=true), a scalarCOUNT(DISTINCT k)over joins can return a wrong result that grows linearly withparallel_pipeline_task_num(e.g. expected 10, got 30 with 3 tasks).The bad plan shape:
AggregationNode.enforceAndDeriveLocalExchangegave a NoRequire distribution to a finalize merge agg with no group keys, treating it likeCOUNT(*). UnlikeCOUNT(*), amulti_distinct_countfinalize agg emits per-instance scalar values that the parentsum0adds up — correctness requires the input to be hash-partitioned by the distinct key. When a PASSTHROUGH local exchange (broadcast-join probe fan-out) scatters same-key rows across instances, the parent double-counts overlapping keys. The result equalscorrect × local task count.The BE-native path was already protected (
AggSinkOperatorX::required_data_distributionchecks_partition_exprs, andchild_breaks_local_key_distributionfrom a prior fix), soenable_local_shuffle_planner=falsewas unaffected — only the FE-planned path was wrong.Root cause
The FE planner used
hasKeys(grouping exprs empty?) as the partition-requirement test, but BE's_partition_exprsis non-empty whenever the agg has group keys or DISTINCT aggregates (distribute_expr_lists+has_distinct). The FE fell back to NoRequire for the distinct case, skipping the hash local exchange that the agg needs.Changes
AggregationNodenow mirrors BE's_partition_exprssemantics viahasPartitionRequirement()(grouping exprs ormulti_distinct_*functions): a finalize agg with a partition requirement demands HASH from its child; only partition-less aggs (COUNT(*)-style) keep NoRequire.satisfy()check passes and no LE is inserted, so the common case is unchanged and free.requiresShuffleForCorrectness()now covers DISTINCT aggregates to match BE'sis_shuffled_operator().Tests
LocalShuffleNodeCoverageTest: unit coverage forAggregationNodeacross finalize/LOCAL/FIRST_MERGE phases × distinct/no-distinct ×enable_local_exchange_before_aggon/off, plusrequiresShuffleForCorrectnesscases. Pre-fix the distinct-finalize case asserted NoRequire; post-fix it asserts RequireHash.LocalExchangePlannerTest: sql-level distributed-plan test — the RQG-shaped query (count(distinct)over a shuffle join + a broadcast join with probe forced to PASSTHROUGH) must contain aLOCAL_EXECUTION_HASH_SHUFFLElocal exchange below the distinct finalize agg. Verified this test fails without the fix (plan only hasPASSTHROUGH) and passes with it.parallel_pipeline_task_num) now returns 10 under all session-var combinations, includingparallel_pipeline_task_num=1/2/4/6.