Fix quadratic aggregation in order with the prealloc-serialized method

[alexey-milovidov]

Aggregation in order (AggregatingInOrderTransform) with a multi-column numbers/strings GROUP BY key, where the data is sorted by a proper prefix of that key, was quadratic in the block size and did not respect max_execution_time. It surfaced as a "Hung check failed, possible deadlock found" in the server-side AST fuzzer.

When the sort order is only a prefix of the GROUP BY key, consume splits the block into runs of equal prefix values and aggregates each run with a hash table over the full key. A multi-column numbers/strings key selects the prealloc_serialized method, whose state serializes all of the block's keys up front on construction. Because a fresh state is built for every run (once per row in the worst case, when the prefix is highly selective), a single block became O(number_of_runs * block_size). And since one consume call processes a whole block without yielding, the query also ignored max_execution_time — which is what tripped the hung check.

The fix makes aggregation in order use the plain serialized method, whose construction is O(1) and which serializes keys lazily per row, restoring linear time. The prealloc_serialized method is kept for whole-block aggregation, where it is a win. The serialized keys are byte-identical between the two methods, so merging across pipeline stages is unaffected.

Reproducer (before the fix this runs for many minutes; after it is instant):

CREATE TABLE t (s String, n UInt64) ENGINE = MergeTree ORDER BY s;
INSERT INTO t SELECT toString(number), number FROM numbers(300000);
SELECT s, n FROM t GROUP BY s, n
SETTINGS optimize_aggregation_in_order = 1, optimize_read_in_order = 1, max_block_size = 16384;

Verified locally: the reproducer above (N = 40000, single block) dropped from ~76 s to ~0.1 s under an ASan build, scales linearly, and produces results identical to regular aggregation across single/multi-key, nullable, prefix-sorted and full-sorted shapes; existing 02233_optimize_aggregation_in_order_prefix* and 01291_aggregation_in_order tests still pass.

No existing tracking issue was found for this failure.

CI report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?REF=master&sha=81f15640fa009593bf323fc5b4259ee834143073&name_0=MasterCI&name_1=Stress%20test%20%28amd_tsan%29

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fixed a quadratic-time blowup (and unresponsiveness to max_execution_time) in aggregation in order when grouping by multiple keys whose sort order is only a prefix of the grouping key.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

Version info

• Merged into: 26.7.1.866 (included in 26.7 and later)

[clickhouse-gh]

Workflow [PR], commit [c137b39]

Summary: ✅

• Performance Comparison: Performance dashboard

---

AI Review

Summary

This PR narrows the prealloc_serialized fallback to the per-run aggregation-in-order path and adds focused regression coverage for the single-stream, multi-stream, and nullable-key cases. I did not find any remaining correctness or contract regressions on the current PR head: the earlier merge-path and nullable-coverage review points are addressed, the new method_chosen_for_in_order split is consistent with the unchanged whole-block merge path, and the current PR CI is green.

Final Verdict

✅ No new findings on the current PR head.

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	85.80%	85.80%	+0.00%
Functions	92.70%	92.70%	+0.00%
Branches	78.00%	78.00%	+0.00%

Changed lines: Changed C/C++ lines covered: 19/20 (95.00%) · Uncovered code

Full report · Diff report

[alexey-milovidov]

It's perfect.