bench: parquet scan with a table schema narrower than a nested column#23397
Open
adriangb wants to merge 1 commit into
Open
bench: parquet scan with a table schema narrower than a nested column#23397adriangb wants to merge 1 commit into
adriangb wants to merge 1 commit into
Conversation
Registers the same wide list<struct> / struct parquet file with the
file's full schema and with a narrower declared schema, plus a
physically-narrow file as the floor. On main the narrow declared
schema reads exactly as many bytes as the full schema (the whole
column is fetched and clipped in memory by the adapter-inserted cast):
list_struct_narrow_schema: bytes_scanned=25.19 MB
list_struct_full_schema: bytes_scanned=25.19 MB
list_struct_physically_narrow: bytes_scanned=3.32 KB
select_events_narrow_schema 3.45 ms
select_events_full_schema 3.36 ms
select_events_physically_narrow 164 µs
Baseline for schema-driven nested projection pruning
(see apache/datafusion-comet#4859).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
When a table's declared schema is narrower than a parquet file's nested column (logical
events: LIST<STRUCT<x, y>>over a physicalLIST<STRUCT<x, y, +8 pads>>), the reader currently fetches and decodes every leaf of the column and discards the extra subfields in memory via the adapter-inserted cast. This is how engines like Spark (via Comet) communicate nested projection pruning to the scan — as a clipped read schema — and it is where Comet measured reading 1.35 TB where Spark read 30.9 GB for the same prunedReadSchema.This PR adds a benchmark that documents the current behavior as a checked-in baseline, independent of any fix:
What changes are included in this PR?
A criterion benchmark,
datafusion/core/benches/parquet_nested_schema_pruning.rs, that registers the same widelist<struct>(and top-level struct) parquet file with both its full schema and a narrower declared schema, plus a physically-narrow file as the floor, and prints the scans'bytes_scannedat setup so the IO pattern is visible alongside wall time.Are these changes tested?
It is a benchmark; it compiles under
cargo bench --no-runand runs green.Are there any user-facing changes?
No.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KuMaRtFSPDQesuzjN5Koyd