Skip to content

Cherry-pick-cd28203e3 - perf: array_agg() performance improvements (#23716) - #161

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
branch-54from
cherry-pick-cd28203e3
Jul 29, 2026
Merged

Cherry-pick-cd28203e3 - perf: array_agg() performance improvements (#23716)#161
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
branch-54from
cherry-pick-cd28203e3

Conversation

@fred1268

Copy link
Copy Markdown

Queries using array_agg(DISTINCT col) were significantly slower than expected.

Profiling revealed that DistinctArrayAggAccumulator::update_batch was allocating a
heap-owned String on every single input row — even for rows whose value was already
present in the accumulator. For a typical low-cardinality workload (e.g. a column of ~25
database names across thousands of rows), this meant paying the full allocation cost for
every duplicate, which dominated the runtime.

This PR applies the same deduplication strategy already used by AggregateExec for
GROUP BY: duplicate rows now cost only a hash probe with no heap allocation, and new
distinct values are appended to a single shared buffer rather than allocated individually.
The fix applies to all column types, not just strings, and retract_batch support
(required for sliding window frames such as ROWS BETWEEN N PRECEDING AND CURRENT ROW)
is fully preserved.

Four unit tests were added to DistinctArrayAggAccumulator — one each for Utf8,
Int64, Float64, and Date32 — to pin the deduplication contract across the most
common column types and serve as a regression guard for future changes. The existing
sliding window sqllogictest suite (array_agg_sliding_window.slt) covers retract_batch
correctness end-to-end and passes unchanged.

Two update_batch micro-benchmarks were added to measure the before/after on realistic
data: one with low cardinality (~25 distinct database names in 8 192 rows, modelling the
common production case) and one with high cardinality (~7 800 distinct values, modelling
the worst case where almost every row is new). Results on an 8 192-row batch:

Benchmark Before After Speedup
Low cardinality (~25 distinct DB names) 648.6 µs 189.4 µs 3.42×
High cardinality (~7 800 distinct values) 1078.3 µs 269.4 µs 4.00×

No user-facing changes

No breaking changes to public APIs


Which issue does this PR close?

  • Closes #.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

- Closes apache#23715.

Queries using `array_agg(DISTINCT col)` were significantly slower than
expected.

Profiling revealed that `DistinctArrayAggAccumulator::update_batch` was
allocating a
heap-owned `String` on every single input row — even for rows whose
value was already
present in the accumulator. For a typical low-cardinality workload (e.g.
a column of ~25
database names across thousands of rows), this meant paying the full
allocation cost for
every duplicate, which dominated the runtime.

This PR applies the same deduplication strategy already used by
`AggregateExec` for
`GROUP BY`: duplicate rows now cost only a hash probe with no heap
allocation, and new
distinct values are appended to a single shared buffer rather than
allocated individually.
The fix applies to all column types, not just strings, and
`retract_batch` support
(required for sliding window frames such as `ROWS BETWEEN N PRECEDING
AND CURRENT ROW`)
is fully preserved.

Four unit tests were added to `DistinctArrayAggAccumulator` — one each
for `Utf8`,
`Int64`, `Float64`, and `Date32` — to pin the deduplication contract
across the most
common column types and serve as a regression guard for future changes.
The existing
sliding window sqllogictest suite (`array_agg_sliding_window.slt`)
covers `retract_batch`
correctness end-to-end and passes unchanged.

Two `update_batch` micro-benchmarks were added to measure the
before/after on realistic
data: one with low cardinality (~25 distinct database names in 8 192
rows, modelling the
common production case) and one with high cardinality (~7 800 distinct
values, modelling
the worst case where almost every row is new). Results on an 8 192-row
batch:

| Benchmark | Before | After | Speedup |
|---|---|---|---|
| Low cardinality (~25 distinct DB names) | 648.6 µs | 189.4 µs |
**3.42×** |
| High cardinality (~7 800 distinct values) | 1078.3 µs | 269.4 µs |
**4.00×** |

No user-facing changes

No breaking changes to public APIs

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit b632fb7 into branch-54 Jul 29, 2026
63 of 66 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the cherry-pick-cd28203e3 branch July 29, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant