Add metric category filtering for EXPLAIN ANALYZE by adriangb · Pull Request #21160 · apache/datafusion

adriangb · 2026-03-25T22:14:11Z

Summary

Adds MetricCategory enum (Rows, Bytes, Timing) classifying metrics by what they measure and, critically, their determinism: rows/bytes are deterministic given the same plan+data; timing varies across runs.
Each Metric can now declare its category via MetricBuilder::with_category(). Well-known builder methods (output_rows, elapsed_compute, output_bytes, etc.) set the category automatically. Custom counters/gauges default to "always included".
New session config datafusion.explain.analyze_categories accepts all (default), none, or comma-separated rows, bytes, timing.
This is orthogonal to the existing analyze_level (summary/dev) which controls verbosity.

Motivation

Running EXPLAIN ANALYZE in .slt tests currently requires liberal use of <slt:ignore> for every non-deterministic timing metric. With this change, a test can simply:

SET datafusion.explain.analyze_categories = 'rows';
EXPLAIN ANALYZE SELECT ...;
-- output contains only row-count metrics — fully deterministic, no <slt:ignore> needed

In particular, for dynamic filters we have relatively complex integration tests that exist mostly to assert the plan shapes and state of the dynamic filters after the plan has been executed. For example #21059. With this change I think most of those can be moved to SLT tests. I've also wanted to e.g. make assertions about pruning effectiveness without having timing information included.

Test plan

New Rust integration test explain_analyze_categories covering all combos (rows, none, all, rows+bytes)
New .slt tests in explain_analyze.slt for rows, none, rows,bytes, and rows with dev level
Existing explain_analyze integration tests pass (24/24)
Proto roundtrip test updated and passing
information_schema slt updated for new config entry
Full core_integration suite passes (918 tests)

🤖 Generated with Claude Code

2010YOUY01 · 2026-03-26T01:25:33Z

datafusion/common/src/format.rs

+    ///
+    /// **Non-deterministic** — varies across runs even on the same hardware.
+    Timing,
+}


Perhaps we can add another variant for all uncategorized metrics, so we can do

set datafusion.explain.analyze_category = 'rows, bytes, uncategorized' -- Only exclude `Timing` metrics category

Introduces `MetricCategory` (Rows, Bytes, Timing) so that EXPLAIN ANALYZE output can be narrowed to only deterministic metrics, which is especially useful in sqllogictest (.slt) files where timing values would otherwise require `<slt:ignore>` markers everywhere. Each `Metric` now optionally declares a category via `MetricBuilder::with_category()`. Well-known builder methods (`output_rows`, `elapsed_compute`, …) set the category automatically; custom counters/gauges default to "always included". A new session config `datafusion.explain.analyze_categories` accepts `all` (default), `none`, or a comma-separated list of `rows`, `bytes`, `timing` to control which categories appear. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use a parquet table with multiple row groups and a TopK ORDER BY LIMIT query that triggers DynamicFilter pushdown. This makes the slt examples much more realistic — they show pruning metrics, row group statistics, and the resolved DynamicFilter predicate. Add a 'timing' category example that shows only elapsed_compute and metadata_load_time (with <slt:ignore> since they are non-deterministic). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Opt in every custom counter/gauge created by DataFusion's own operators (parquet, file_stream, joins, aggregates, topk, unnest, buffer) so that category filtering works cleanly out of the box. For example `bytes_scanned` → Bytes, `pushdown_rows_pruned` → Rows, `peak_mem_used` → Bytes, `row_replacements` → Rows, etc. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

adriangb · 2026-03-27T00:10:58Z

datafusion/common/src/format.rs

 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub enum ExplainAnalyzeLevel {
-    /// Show a compact view containing high-level metrics
+pub enum MetricType {


@2010YOUY01 this is now a bigger diff because I refactored MetricType into here. It removes the redundancy with ExplainAnalyzeLevel (they were the same enum). I really wanted to have MetricCategory in this module (or inside datafusion-common so that it's available to this module) so that we can parse the strings directly into the enum variants and avoid carrying around Vec<String>s. I think this also applies to MetricType. Now both code paths are parallel, there is less duplicate code and validation happens earlier.

One thing to confirm is that we are okay with having MetricType and MetricCategory. They sound somewhat redundant but I think are actually orthogonal, unless we can say something like "all timing metrics are Dev" in which case one is a superset of the other. I'd like your input here since you added MetricType

I agree the new category tags are necessary

2010YOUY01

Thanks, this feature LGTM overall. I have one suggestion in the above comment: consider allowing a metric to have multiple associated categories. Curious to hear your thoughts.

2010YOUY01 · 2026-03-27T05:01:44Z

datafusion/common/src/format.rs

 #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
-pub enum ExplainAnalyzeLevel {
-    /// Show a compact view containing high-level metrics
+pub enum MetricType {


I agree the new category tags are necessary

2010YOUY01 · 2026-03-27T05:27:51Z

datafusion/physical-expr-common/src/metrics/builder.rs

+    ///
+    /// See [`MetricCategory`] for details on the determinism properties
+    /// of each category.
+    pub fn with_category(mut self, category: MetricCategory) -> Self {


I'm wondering if some metrics may need multiple categories.

Currently Timing/Bytes/Rows classify metrics by output type, but we could also group them by functional aspects like Pruning or Spilling. That would enable more flexible filtering, e.g.:

set datafusion.explain.analyze_categories = ['Timing', 'Spilling']

to show metrics matching both tags.

If this direction is plausible, should we consider storing multiple categories (e.g. Vec<MetricCategory>) instead? To be more flexible for the future change.

Could we add that as a followup feature? It wouldn't be breaking I think.

That makes sense, if it won't need breaking changes.

2010YOUY01 reviewed Mar 26, 2026

View reviewed changes

adriangb and others added 7 commits March 26, 2026 15:16

fix

701aab9

fix configs

c290b2a

fixes

010dc6c

update json

b13a270

adriangb force-pushed the explain-analyze-metric-categories branch from 90963d4 to b13a270 Compare March 26, 2026 20:26

github-actions bot added the documentation Improvements or additions to documentation label Mar 26, 2026

refactor

5cf7348

adriangb force-pushed the explain-analyze-metric-categories branch from dc72f7a to 5cf7348 Compare March 26, 2026 23:38

adriangb commented Mar 27, 2026

View reviewed changes

2010YOUY01 reviewed Mar 27, 2026

View reviewed changes

2010YOUY01 approved these changes Mar 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metric category filtering for EXPLAIN ANALYZE#21160

Add metric category filtering for EXPLAIN ANALYZE#21160
adriangb wants to merge 8 commits intoapache:mainfrom
pydantic:explain-analyze-metric-categories

adriangb commented Mar 25, 2026 •

edited

Loading

Uh oh!

2010YOUY01 Mar 26, 2026

Uh oh!

adriangb Mar 27, 2026

Uh oh!

adriangb Mar 27, 2026 •

edited

Loading

Uh oh!

2010YOUY01 Mar 27, 2026

Uh oh!

2010YOUY01 left a comment

Uh oh!

2010YOUY01 Mar 27, 2026

Uh oh!

2010YOUY01 Mar 27, 2026

Uh oh!

adriangb Mar 27, 2026

Uh oh!

2010YOUY01 Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

adriangb commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Test plan

Uh oh!

2010YOUY01 Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adriangb commented Mar 25, 2026 •

edited

Loading

adriangb Mar 27, 2026 •

edited

Loading