Refine and document AggregateExec metrics - #24757
Open
kosiew wants to merge 15 commits into
Open
Conversation
… mode scope, dev visibility, EXPLAIN example. - Added comprehensive coverage of AggregateExec metrics, including new counters, gauges, and timing measurements. - Documented the distinct metrics **phases** (e.g., parse, optimize, execute) and their significance in performance analysis. - Introduced the **indexes/labels** schema used to tag metrics for efficient querying and grouping. - Clarified the **mode scope** (global vs. session) and how it influences metric collection and visibility. - Expanded the **dev visibility** section with guidelines for developers to monitor and interpret these metrics. - Included a practical **EXPLAIN** example that demonstrates how the new metrics appear in query output.
…hase, indexing, labels, and FILTER handling
- Replace terse description with structured table of metrics - Add details on per‑aggregate timers and phase availability - Clarify which metrics are reported for different aggregation modes and implementations - Update description of time boundaries between `time_calculating_group_ids` and `aggregation_time`
- Correct group-ID metric wording/gates. - Add peak_mem_used. - Caveat legacy aggregation_time inflation. - Clarify partial-stage timer example. - Explain Dev vs Summary + config key.
- Refined the TopK timer scope description to clarify precise measurement semantics, removing any ambiguity about timing granularity. - Added explicit documentation for the legacy skip path, detailing its conversion to state via the `convert_to_state` function and providing clear usage examples.
- Correct number of untimed group-key work paths from two to three - Specify that grouping-expression evaluation is not timed for Grouped TopK - Clarify that per-aggregate timers carry an `aggregate` label, visible only in EXPLAIN ANALYZE VERBOSE
…regate timers - Updated `docs/source/user-guide/metrics.md`: - Added explicit description of the scope for TopK untimed group‑expression, clarifying when and how the expression applies. - Revised the label scope section to specify that labels are limited to per‑aggregate timers only.
…d prefix-sum aggregation - Unified grouped timer boundaries. - Fixed legacy aggregation prefix-sum. - Added TopK maintenance metric; omits accumulator metric. - Covered empty grouping-set timer bypasses. - Updated metrics documentation. - Added TopK metric regression assertions.
- Updated the `metrics.md` documentation to clearly distinguish between normal-output and memory-pressure states. - Added explicit timing descriptions for each state, explaining when each occurs and how long it typically lasts. - Refined wording throughout the section to improve readability and remove ambiguity. - Included illustrative examples that contrast the timing behavior of normal-output and memory-pressure states. - Adjusted any related references to ensure consistent terminology across the user guide.
…nd state materialization - Central GroupByMetrics scoped APIs; timer fields private. - TopK metric owned centrally; removed `aggregation_time`. - Migrated aggregate paths to new structure. - State materialization now included in `emitting_time`. - Updated documentation. - Added test for no‑accumulator metric registration.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24757 +/- ##
==========================================
+ Coverage 81.47% 81.51% +0.03%
==========================================
Files 1122 1123 +1
Lines 404140 405263 +1123
Branches 404140 405263 +1123
==========================================
+ Hits 329284 330340 +1056
- Misses 55546 55584 +38
- Partials 19310 19339 +29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- Adjusted the documentation label to accurately reflect the updated terminology. - Changed the function call from `sum(t.a)` to `SUM(a)` to align with the new naming convention. - Ensured consistency between the documentation and the implementation. - Verified that the changes resolve any labeling ambiguities in the documentation. - Updated any related references to use the new label and function name.
…TopK - Added caching of the `is_stream_merging` flag per batch to avoid recomputation on every operation. - Added caching of the `is_group_by_only` flag for each TopK internal call, reducing redundant checks. - Updated the relevant internal methods to store and reuse these cached values, improving performance. - Ensured cache invalidation is synchronized with batch and TopK lifecycle to maintain correctness.
…cked grouped paths, clarify non‑grouped timing and DISTINCT TopK bypass, fix rendered‑label example, and add related tests - Docs scope emitting_time to accumulator‑backed grouped paths. - Docs explain non‑grouped timing + limited DISTINCT TopK bypass. - Corrected rendered‑label example: `sum(t.a)`. - Tests: non‑grouped lacks emitting_time; limited DISTINCT lacks TopK metric. - Added normal/verbose `EXPLAIN` mapping test for 3 aggregates.
…roup values and accumulator results, removing claim of full output‑batch production - Updated the description of `emitting_time` to indicate it materializes group values and accumulator results. - Clarified that the process does not guarantee full output‑batch production. - Adjusted wording to avoid misleading implications about batch completeness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Aggregate-specific metrics are a user-visible observability surface in
EXPLAIN ANALYZE, but their phase boundaries and relationship to individual aggregate expressions need to be clear and consistent.In particular, users should be able to understand what each aggregate timer measures, map indexed metrics such as
agg_expr_0_arguments_timeback to the corresponding expression in the operator'saggr=[...]list, and understand which metrics are available on different grouped aggregation paths without reading the implementation.This PR also tightens the timing boundaries of the grouped aggregate metrics so that the documented metrics correspond more directly to the work they describe.
What changes are included in this PR?
Documents
AggregateExecoperator-level and per-aggregate metrics, including their phase boundaries, indexed naming convention, aggregate expression labels, visibility, and path-specific coverage.Groups aggregate timing into explicit phases for:
update/mergeoperations, andEnsures accumulator
aggregation_timemeasures a single interval containing the relevant accumulator operations rather than accumulating overlapping/prefix timing.Includes group interning and ordering setup in
time_calculating_group_ids.Includes collectively evaluated aggregate filters in
aggregate_arguments_time.Adds
topk_maintenance_timefor Grouped TopK priority-map maintenance and avoids registeringaggregation_timefor that accumulator-free path.Times grouped output materialization consistently with
emitting_time.Adds
EXPLAIN ANALYZEcoverage demonstrating how indexed per-aggregate metrics map to multiple aggregate expressions and how expression labels appear in verbose output.Are these changes tested?
Yes.
The patch adds and updates tests that verify:
EXPLAIN ANALYZEreportsagg_expr_0_arguments_time,agg_expr_1_arguments_time, andagg_expr_2_arguments_timefor a query containingSUM(c5),SUM(c6), andCOUNT(c7).EXPLAIN ANALYZE VERBOSEassociates those indexed metrics with the corresponding rendered aggregate expressions.topk_maintenance_timeandtime_calculating_group_idswhile not registeringaggregation_time.topk_maintenance_time.emitting_time.GroupByMetrics::new_topkdoes not registeraggregation_time.Are there any user-facing changes?
Yes. Aggregate metrics shown through
EXPLAIN ANALYZEare clarified and their timing boundaries are made more consistent.The metrics documentation now describes the
AggregateExecoperator-level metrics and per-expression timer phases, explains how indexed metrics map to aggregate expressions, and documents path-specific differences such as Grouped TopK and non-grouped aggregation.These are observability and documentation changes; this PR does not introduce a public API change.
LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.