Skip to content

Add per-aggregate accumulator phase metrics - #24423

Draft
kosiew wants to merge 6 commits into
apache:mainfrom
kosiew:aggmetrics-02-23570
Draft

Add per-aggregate accumulator phase metrics#24423
kosiew wants to merge 6 commits into
apache:mainfrom
kosiew:aggmetrics-02-23570

Conversation

@kosiew

@kosiew kosiew commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Migrated grouped hash aggregation currently exposes timing for aggregate argument evaluation and shared aggregation/emission work, but does not show how much time each aggregate expression spends performing accumulator operations.

This makes it difficult to identify aggregates where the accumulator itself is the source of execution cost. This PR adds per-aggregate-expression timing for the update, merge, state, and evaluate phases used by migrated hash aggregation.

What changes are included in this PR?

  • Adds per-aggregate accumulator timing metrics for update, merge, state, and evaluate phases.
  • Creates the applicable phase metrics based on the aggregate mode.
  • Instruments accumulator operations in both regular and ordered migrated aggregate tables.
  • Covers partial, partial-reduce, final, and single aggregation paths, including state emission and spill/replay paths.
  • Reuses aggregate-expression labels so metrics from different aggregate expressions can be distinguished.
  • Keeps the existing group-by, aggregate argument, aggregation_time, and emitting_time metrics.
  • Does not change the public GroupsAccumulator API.

The new metrics use names such as agg_expr_0_update_time, agg_expr_0_merge_time, agg_expr_0_state_time, and agg_expr_0_evaluate_time, with the corresponding aggregate expression attached as the aggregate label.

Are these changes tested?

Yes.

The tests in this PR:

  • Add accumulator_phases_match_aggregate_mode to verify the accumulator phases configured for all six aggregate modes.
  • Extend the aggregate metrics tests to verify separate SUM(a) and SUM(b) labels for arguments_time, update_time, and state_time, and verify positive update/state timings.
  • Extend the final aggregate metrics test to verify merge_time and evaluate_time for SUM(b) and verify that those timings are positive.
  • Explicitly enable datafusion.execution.enable_migration_aggregate in the relevant metrics tests.
  • Extend migrated aggregate execution tests with assert_accumulator_phase_times to verify positive update/state timings for partial aggregation and merge/evaluate timings for final aggregation.
  • Update the spilling single-aggregate test to use a migrated spill context and verify positive update, state, merge, and evaluate timings.

Are there any user-facing changes?

Yes, for users inspecting execution metrics. Migrated hash aggregation now exposes per-aggregate-expression accumulator phase timing metrics, making it possible to distinguish accumulator work from aggregate argument evaluation and shared aggregation/emission work.

There are no changes to aggregate query semantics or public accumulator APIs.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

kosiew added 6 commits August 17, 2026 14:06
…ables

- Introduce per-agg timers for update/merge/state/evaluate phases.
- Use hash and ordered tables for migrated data.
- Expose only phase-valid metrics.
- Add timing for partial-skip state conversion.
- Include tests for partial and final metric labels.
- Central accumulator_phases(mode) helper.
- Removed duplicated mode→phase matches.
- Added all-mode unit test.
…le modes, cover partitioned variants, and add regression test for forced‑spill ordered‑single
- Adds positive-duration assertions for partial update/state
- Adds positive-duration assertions for final merge/evaluate
- Retains existing exact label/name checks
- References ordered spill/replay coverage added in prior commit
- Centralize hash/ordered metric setup via a private AggregateTableMetrics factory.
- Preserve Arc sharing in `from_hash_table`.
- Add a table‑driven phase matrix test that retains all 6 modes.
…inal aggregates

- Ordered partial now asserts that update.state is positive in update.
- Ordered final now asserts that merge is positive in merge evaluation.
- Shared test helper introduced to reduce duplication.
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 17, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.74468% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.24%. Comparing base (9a83691) to head (84120c2).

Files with missing lines Patch % Lines
...c/aggregates/aggregate_hash_table/partial_table.rs 73.91% 6 Missing ⚠️
.../aggregates/aggregate_hash_table/common_ordered.rs 90.62% 1 Missing and 2 partials ⚠️
datafusion/physical-plan/src/aggregates/mod.rs 90.32% 0 Missing and 3 partials ⚠️
...plan/src/aggregates/aggregate_hash_table/common.rs 92.30% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #24423    +/-   ##
========================================
  Coverage   81.23%   81.24%            
========================================
  Files        1112     1113     +1     
  Lines      390627   390894   +267     
  Branches   390627   390894   +267     
========================================
+ Hits       317337   317584   +247     
- Misses      54656    54671    +15     
- Partials    18634    18639     +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants