Blog post about efficient filter representation in Parquet filter pushdown

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• part of [Epic] Parquet Reader Improvement Plan / Proposal - July 2025 #8000
• Related to [Parquet] Adaptive Parquet Predicate Pushdown #8733 from @hhhizzz
• Related to Adaptive Parquet Predicate Pushdown Evaluation #5523

TLDR is I want to 1) advance the state of understanding of how late materialization / filter pushdown works, and 2) tell the world how great the Rust implementation is (and implicitly explain what other types of optimizations are unlocked by this)

I think there is significant room to help industrial practitioners by explaining the challenges that go into implementing late materialization "for real" in an industrial strength Parquet reader

Background

The techniques for implementing "late materialization" in column stores is well understood and explained well first in 2006/2007:

• Materialization Strategies in a Column-Oriented DBMS
• Column-Stores vs. Row-Stores: How Different Are They Really?

The current Rust Parquet reader supports late materialization (basically the "EM Pipelined" strategy in this diagram from Materialization Strategies in a Column-Oriented DBMS

The API for evaluating predicates during the scan via the ArrowReaderBuilder::with_row_filter. See details on the RowFilter API .

@XiangpengHao also gives a good background treatment in the context of adding a predicate cache (to avoid the overhead of decompressing pages twice): https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown/

However, it has taken us several years (and we are still not quite there) to get to the point that we can turn on late materialization on "for real" due to various engineering challenges (decompression speed).

Interestingly, there is a similar discussion on filter representation in Predicate Caching: Query-Driven Secondary Indexing for Cloud Data Warehouses -- referred to as 4.1.1 Range Index and 4.1.2 Bitmap Index

Describe the solution you'd like

I would like to write a blog that highlights the tradeoffs in filter representation how we worked to improve it.

Describe alternatives you've considered

Additional context

[hhhizzz]

I would be honored if this blog post could be written by me. I will do my best to submit a first draft this week. I don't yet fully understand the overall design of filter pushdown, and I look forward to everyone pointing out issues in the draft and helping to improve it.

[alamb]

That sounds great @hhhizzz -- thank you . Please let us know how we can help (in addition to reviewing the draft)

[hhhizzz]

I’d like to give a quick progress update: this week I mainly read the papers mentioned above and relevant arrow-rs code, and tried to draft the approach. I expect it will take a few more days before I can submit the first version.

One question I’d like to ask/confirm: based on my current understanding, the current implementation looks more like an LM (Late Materialization) pipeline rather than an EM (Early Materialization) pipeline. Your text and ref image are both EM, but context is LM.
My understanding is: predicate columns are cached instead of being directly assembled into the tuple; they are read again later when needed, but because of the cache there’s no repeated decoding, so the full decoding process doesn’t have to be run again. I want to confirm whether this understanding is correct, or whether I’m missing some key point? Thanks. @alamb

[alamb]

I’d like to give a quick progress update: this week I mainly read the papers mentioned above and relevant arrow-rs code, and tried to draft the approach. I expect it will take a few more days before I can submit the first version.

Amazing!

One question I’d like to ask/confirm: based on my current understanding, the current implementation looks more like an LM (Late Materialization) pipeline rather than an EM (Early Materialization) pipeline. Your text and ref image are both EM, but context is LM.

Yes, I am sorry -- I agree that what we have in the parquet reader is an early materialization pipeline

My understanding is: predicate columns are cached instead of being directly assembled into the tuple; they are read again later when needed, but because of the cache there’s no repeated decoding, so the full decoding process doesn’t have to be run again.

Yes, that is correct. Specifically, we (well @XiangpengHao ) found that the cost of re-decoding (specifically decompressing with ZSTD or other block compression) often outweighed the benefits of late materialization