feat(core): metrics() for OrderedRangeRepartitionExec and RuntimeStatsExec - #2250
Merged
avantgardnerio merged 3 commits intoAug 7, 2026
Merged
Conversation
Was returning `None` from `metrics()`, so it was invisible in the scheduler's stage-metrics dump AND caused display-walker misalignment: metrics for later operators (SortExec, DataSourceExec) got labeled as this op or ORRE, because `collect_plan_metrics` skips None returns while the display visitor increments its index for every node. Adds `ExecutionPlanMetricsSet` with: - `elapsed_compute` / `output_rows` via `BaselineMetrics.record_output` - `sketch_time` (subset_time) — isolates the T-Digest `merge_unsorted_f64` cost from surrounding evaluate/downcast/flatten, so sketch-mode overhead is measurable separately. - `sketch_batches` (Count) — number of batches where the sketch was actually updated (excludes empty-after-nulls batches). Timer scoped post-child-poll (Time clone so the timer doesn't hold a borrow across the mutable `ingest` call). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ORRE was constructing BaselineMetrics for its StreamingMerge output side but never exposing them (no `fn metrics()` impl), and the scatter side — where the actual value-add of range-routing lives — had zero instrumentation. Adds `fn metrics()` so both sides surface, plus per-input-partition scatter counters: - `scatter_elapsed_compute` — total scatter compute per input, timer scoped post child-poll (upstream sort/read not billed). - `scatter_split_time` — subset time for `split_batch_by_range` alone. Isolates routing-expr evaluation cost from the surrounding channel work. - `scatter_send_time` — subset time for `senders[out].send().await`. High values relative to compute mean downstream merge is draining slower than we can scatter (backpressure). - `scatter_discover_cuts_time` — one-shot first-batch cost of the cut-discovery walk into RuntimeStatsExec. - `scatter_input_batches` / `scatter_input_rows` — input volume. - `scatter_output_sub_batches` — post-split fanout count. Under skew this can be up to K× input_batches. Compute timer is stopped around `send().await` so backpressure waits land in `send_time` alone, not double-counted into elapsed_compute. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tter fn Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@phillipleblanc now that I'm using these for real, I needed metrics to debug performance. |
milenkovicm
approved these changes
Aug 7, 2026
milenkovicm
left a comment
Contributor
There was a problem hiding this comment.
thanks @avantgardnerio
| } | ||
|
|
||
| /// Every input row is emitted exactly once. Overrides default `Unknown`. | ||
| fn metrics(&self) -> Option<MetricsSet> { |
Contributor
There was a problem hiding this comment.
it happened to me as well (not once) 😀
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Both
OrderedRangeRepartitionExecandRuntimeStatsExecwere returningNonefrommetrics(), so they were invisible in the scheduler's stage-metrics dump. Beyond just missing signal, this caused a display-walker misalignment insideRuntimeStatsExec:collect_plan_metricsskipsNonereturns while the display visitor increments its index for every node, so metrics for later operators (SortExec, DataSourceExec) got labeled as the wrong op.Both ops are additive here — no behavior change, no execution-path change, just wiring in the standard
ExecutionPlanMetricsSetand returning it frommetrics().RuntimeStatsExecAdds
ExecutionPlanMetricsSetwith:elapsed_compute/output_rowsviaBaselineMetrics.record_outputsketch_time(subset time) — isolates the T-Digestmerge_unsorted_f64cost from surrounding evaluate/downcast/flatten, so sketch-mode overhead is measurable separately.sketch_batches(Count) — number of batches where the sketch was actually updated (excludes empty-after-nulls batches).Timer scoped post-child-poll so upstream shuffle IO / parquet reads aren't billed to
elapsed_compute.OrderedRangeRepartitionExecOrderedRangeRepartitionExecwas constructingBaselineMetricsfor itsStreamingMergeoutput side but never exposing them (nometrics()impl), and the scatter side — where the actual value-add of range-routing lives — had zero instrumentation.Adds
fn metrics()so the merge-side baseline surfaces, plus per-input-partition scatter counters:scatter_elapsed_compute— total scatter compute per input, timer scoped post child-poll.scatter_split_time— subset time forsplit_batch_by_rangealone. Isolates routing-expr evaluation cost from the surrounding channel work.scatter_send_time— subset time forsenders[out].send().await. High values relative to compute mean downstream merge is draining slower than we can scatter (backpressure).scatter_discover_cuts_time— one-shot first-batch cost of the cut-discovery walk intoRuntimeStatsExec.scatter_input_batches/scatter_input_rows— input volume.scatter_output_sub_batches— post-split fanout count. Under skew this can be up to K× input_batches.Compute timer is stopped around
send().awaitso backpressure waits land insend_timealone, not double-counted intoelapsed_compute.Context
Extracted from #2223 (parallel BWAG for the range-window shape). Both ops already exist on main from prior slices; this is pure instrumentation on top of them, standalone.