perf: prune window state only for partitions that made progress - #24148
Open
neilconway wants to merge 1 commit into
Open
perf: prune window state only for partitions that made progress#24148neilconway wants to merge 1 commit into
neilconway wants to merge 1 commit into
Conversation
Contributor
Author
|
FYI @Dandandan @avantgardnerio @2010YOUY01 -- further work on optimizing window function evaluation for workloads with many partitions |
neilconway
force-pushed
the
neilc/perf-window-prune-bookkeeping
branch
from
August 6, 2026 16:28
03ae9bd to
992e5d6
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24148 +/- ##
==========================================
- Coverage 81.05% 81.05% -0.01%
==========================================
Files 1106 1106
Lines 380556 380567 +11
Branches 380556 380567 +11
==========================================
+ Hits 308477 308485 +8
- Misses 53861 53862 +1
- Partials 18218 18220 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
BoundedWindowAggExecinLinearmode is slow for many-partitions #23982Rationale for this change
After ingesting a batch of data, updating accumulator state, and emitting new output rows,
BoundedWindowAggStreamprunes each partition to reclaim state that is no longer needed:prune_out_columnstrims emitted results that are no longer needed, andprune_partition_batchesdrops buffered input rows that aren't needed by any window expression.Both functions did work proportional to the # of live partitions, despite pruning being a no-op for partitions that didn't receive rows in the most recent batch:
prune_out_columnslooked up every partition's buffer by hashing its partition key and re-sliced every result column, including zero-length prunes that rebuilt an identical column.prune_partition_batchesput an entry in its prune-count map for every live partition, cloning each partition's key (a Vec); for sparse workloads (# of partitions > batch-size), most prune counts will be zero and this did a lot of redundant work.Restructure both passes to pass over quiet partitions:
prune_out_columnsiterates the partition buffers and only processes partitions with a nonzero emitted-row count. Hash lookups now happen only for partitions that emitted rows since the previous pass.prune_partition_batchesonly keeps partitions with positive prune counts in its mapBenchmarks (after applying #24127):
What changes are included in this PR?
Are these changes tested?
Yes, covered by existing tests.
Are there any user-facing changes?
No.