Cherry-pick-cd28203e3 - perf: array_agg() performance improvements (#23716) - #161
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits intoJul 29, 2026
Merged
Conversation
- Closes apache#23715. Queries using `array_agg(DISTINCT col)` were significantly slower than expected. Profiling revealed that `DistinctArrayAggAccumulator::update_batch` was allocating a heap-owned `String` on every single input row — even for rows whose value was already present in the accumulator. For a typical low-cardinality workload (e.g. a column of ~25 database names across thousands of rows), this meant paying the full allocation cost for every duplicate, which dominated the runtime. This PR applies the same deduplication strategy already used by `AggregateExec` for `GROUP BY`: duplicate rows now cost only a hash probe with no heap allocation, and new distinct values are appended to a single shared buffer rather than allocated individually. The fix applies to all column types, not just strings, and `retract_batch` support (required for sliding window frames such as `ROWS BETWEEN N PRECEDING AND CURRENT ROW`) is fully preserved. Four unit tests were added to `DistinctArrayAggAccumulator` — one each for `Utf8`, `Int64`, `Float64`, and `Date32` — to pin the deduplication contract across the most common column types and serve as a regression guard for future changes. The existing sliding window sqllogictest suite (`array_agg_sliding_window.slt`) covers `retract_batch` correctness end-to-end and passes unchanged. Two `update_batch` micro-benchmarks were added to measure the before/after on realistic data: one with low cardinality (~25 distinct database names in 8 192 rows, modelling the common production case) and one with high cardinality (~7 800 distinct values, modelling the worst case where almost every row is new). Results on an 8 192-row batch: | Benchmark | Before | After | Speedup | |---|---|---|---| | Low cardinality (~25 distinct DB names) | 648.6 µs | 189.4 µs | **3.42×** | | High cardinality (~7 800 distinct values) | 1078.3 µs | 269.4 µs | **4.00×** | No user-facing changes No breaking changes to public APIs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
gh-worker-dd-mergequeue-cf854d
Bot
merged commit Jul 29, 2026
b632fb7
into
branch-54
63 of 66 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
array_agg()performance apache/datafusion#23715.Queries using
array_agg(DISTINCT col)were significantly slower than expected.Profiling revealed that
DistinctArrayAggAccumulator::update_batchwas allocating aheap-owned
Stringon every single input row — even for rows whose value was alreadypresent in the accumulator. For a typical low-cardinality workload (e.g. a column of ~25
database names across thousands of rows), this meant paying the full allocation cost for
every duplicate, which dominated the runtime.
This PR applies the same deduplication strategy already used by
AggregateExecforGROUP BY: duplicate rows now cost only a hash probe with no heap allocation, and newdistinct values are appended to a single shared buffer rather than allocated individually.
The fix applies to all column types, not just strings, and
retract_batchsupport(required for sliding window frames such as
ROWS BETWEEN N PRECEDING AND CURRENT ROW)is fully preserved.
Four unit tests were added to
DistinctArrayAggAccumulator— one each forUtf8,Int64,Float64, andDate32— to pin the deduplication contract across the mostcommon column types and serve as a regression guard for future changes. The existing
sliding window sqllogictest suite (
array_agg_sliding_window.slt) coversretract_batchcorrectness end-to-end and passes unchanged.
Two
update_batchmicro-benchmarks were added to measure the before/after on realisticdata: one with low cardinality (~25 distinct database names in 8 192 rows, modelling the
common production case) and one with high cardinality (~7 800 distinct values, modelling
the worst case where almost every row is new). Results on an 8 192-row batch:
No user-facing changes
No breaking changes to public APIs
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?