Skip to content

Teach zoned pruning to lower StatFn#7937

Open
gatesn wants to merge 1 commit into
ngates/stats-7707/built-in-rewrite-rulesfrom
ngates/stats-7707/zoned-statfn-pruning
Open

Teach zoned pruning to lower StatFn#7937
gatesn wants to merge 1 commit into
ngates/stats-7707/built-in-rewrite-rulesfrom
ngates/stats-7707/zoned-statfn-pruning

Conversation

@gatesn
Copy link
Copy Markdown
Contributor

@gatesn gatesn commented May 14, 2026

Summary

  • lower StatFn-based pruning predicates over zoned stats instead of the legacy StatsCatalog path
  • map null/NaN pruning aggregates to existing NullCount/NaNCount zone stats plus row_count
  • remove the old zoned pruning fallback to checked_pruning_expr

Tests

  • cargo test -p vortex-array stats::rewrite::builtins
  • cargo test -p vortex-array all_nan
  • cargo test -p vortex-array all_non_nan
  • cargo test -p vortex-layout zone_map
  • cargo test -p vortex-layout -- --test-threads=1
  • cargo clippy -p vortex-array --all-targets --all-features
  • cargo clippy -p vortex-layout --all-targets --all-features
  • ./scripts/public-api.sh
  • git diff --check

@gatesn gatesn added the changelog/feature A new feature label May 14, 2026
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from ffbdb5f to 1580a9e Compare May 14, 2026 23:57
@gatesn gatesn mentioned this pull request May 14, 2026
39 tasks
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 15, 2026

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1216 untouched benchmarks


Comparing ngates/stats-7707/zoned-statfn-pruning (096128a) with ngates/stats-7707/built-in-rewrite-rules (ad61595)

Open in CodSpeed

@gatesn gatesn force-pushed the ngates/stats-7707/built-in-rewrite-rules branch from 410ff05 to 5cdafee Compare May 15, 2026 00:15
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 1580a9e to c69b282 Compare May 15, 2026 00:15
@gatesn gatesn force-pushed the ngates/stats-7707/built-in-rewrite-rules branch from 5cdafee to 9700eec Compare May 15, 2026 03:51
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from c69b282 to f9cf88c Compare May 15, 2026 03:51
Comment thread vortex-array/src/aggregate_fn/fns/all_nan/mod.rs
Comment thread vortex-array/src/aggregate_fn/fns/all_non_nan/mod.rs
Comment thread vortex-array/src/scalar_fn/fns/stat.rs Outdated
Comment thread vortex-array/src/stats/rewrite/builtins.rs
Comment thread vortex-array/src/stats/rewrite/builtins.rs
Comment thread vortex-layout/src/layouts/zoned/zone_map.rs
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from f9cf88c to ed63b11 Compare May 15, 2026 08:46
@gatesn gatesn marked this pull request as ready for review May 15, 2026 08:52
@gatesn gatesn added the action/benchmark-sql Trigger SQL benchmarks to run on this PR label May 15, 2026
@gatesn gatesn force-pushed the ngates/stats-7707/built-in-rewrite-rules branch from 9700eec to bc790ff Compare May 15, 2026 09:16
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from ed63b11 to 0d3003c Compare May 15, 2026 09:16
Signed-off-by: Nicholas Gates <nick@nickgates.com>
@gatesn gatesn force-pushed the ngates/stats-7707/built-in-rewrite-rules branch from bc790ff to ad61595 Compare May 15, 2026 09:23
@gatesn gatesn force-pushed the ngates/stats-7707/zoned-statfn-pruning branch from 0d3003c to 096128a Compare May 15, 2026 09:23
@gatesn gatesn added the tldr label May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

action/benchmark-sql Trigger SQL benchmarks to run on this PR changelog/feature A new feature tldr

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant