Skip to content

[datafusion] Align Paimon scan statistics with Parquet - #548

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/tpcds-datafusion-statistics
Jul 19, 2026
Merged

[datafusion] Align Paimon scan statistics with Parquet#548
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/tpcds-datafusion-statistics

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Align Paimon scan statistics with DataFusion native Parquet semantics. This exposes conservative manifest-derived column statistics while avoiding compressed full-file bytes that do not represent projected scan bytes.

Changes

  • decode per-file value statistics using the current schema and dense column mappings
  • aggregate manifest min/max bounds and publish exact null counts only for row-preserving scans
  • fail open on malformed, inconsistent, schema-mismatched, or ordering-incompatible statistics
  • suppress unsafe statistics for aggregation merge engines, filters, limits, row ranges, and deletion files
  • report total_byte_size as unknown because Paimon only stores compressed full-file sizes
  • add regression coverage for aggregation, unsafe null-count paths, malformed rows, and binary ordering

Testing

  • cargo test -p paimon --all-targets --features fulltext,vortex
  • cargo test -p paimon value_stats_for_field
  • cargo test -p paimon-datafusion --lib test_partition_statistics
  • cargo test -p paimon-datafusion --lib test_binary_manifest_bounds
  • cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
  • cargo fmt --all -- --check
  • git diff --check

Runtime filter support remains a separate follow-up.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The manifest-derived bounds are conservative, exact null counts are restricted to row-preserving scans, and malformed or schema-mismatched statistics safely fall back to unknown.

@JingsongLi
JingsongLi merged commit 6e8487b into apache:main Jul 19, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants