Is your feature request related to a problem or challenge?
IcebergTableProvider::supports_filters_pushdown in crates/integrations/datafusion/src/table/mod.rs:163-169 unconditionally returns TableProviderFilterPushDown::Inexact for every filter:
fn supports_filters_pushdown(
&self,
filters: &[&Expr],
) -> DFResult<Vec<TableProviderFilterPushDown>> {
Ok(vec![TableProviderFilterPushDown::Inexact; filters.len()])
}
Inexact tells DataFusion: "the scan may apply this filter, but don't trust it, re-evaluate it above the scan." DataFusion therefore leaves a FilterExec on top of IcebergTableScan that re-evaluates the same predicate on every row the scan emits.
However, the scan does already apply the predicate exactly. In crates/iceberg/src/arrow/reader.rs:244-256, the bound predicate is pushed into the Parquet reader as an Arrow RowFilter (ArrowPredicateFn), so every batch yielded by the scan already satisfies the filter. The FilterExec above is pure overhead.
Reproducer
EXPLAIN ANALYZE output with the query SELECT * FROM t WHERE foo = 1 (a filter that is losslessly convertible to an iceberg Predicate):
CoalesceBatchesExec: ..., metrics=[output_rows=1, elapsed_compute=19µs]
FilterExec: foo@0 = 1, metrics=[output_rows=1, elapsed_compute=90µs] <-- redundant
RepartitionExec: ..., metrics=[fetch_time=4.4ms, ...]
IcebergTableScan predicate:[foo = 1] metrics=[]
FilterExec evaluates foo@0 = 1 a second time on every row the scan emits.
Describe the solution you'd like
In supports_filters_pushdown, return TableProviderFilterPushDown::Exact for filters that convert_filter_to_predicate translates losslessly into an iceberg Predicate, and Inexact (or Unsupported) otherwise.
The non-trivial part is detecting lossy conversions.
Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community
Is your feature request related to a problem or challenge?
IcebergTableProvider::supports_filters_pushdownincrates/integrations/datafusion/src/table/mod.rs:163-169unconditionally returnsTableProviderFilterPushDown::Inexactfor every filter:Inexacttells DataFusion: "the scan may apply this filter, but don't trust it, re-evaluate it above the scan." DataFusion therefore leaves aFilterExecon top ofIcebergTableScanthat re-evaluates the same predicate on every row the scan emits.However, the scan does already apply the predicate exactly. In
crates/iceberg/src/arrow/reader.rs:244-256, the bound predicate is pushed into the Parquet reader as an ArrowRowFilter(ArrowPredicateFn), so every batch yielded by the scan already satisfies the filter. TheFilterExecabove is pure overhead.Reproducer
EXPLAIN ANALYZEoutput with the querySELECT * FROM t WHERE foo = 1(a filter that is losslessly convertible to an icebergPredicate):FilterExecevaluatesfoo@0 = 1a second time on every row the scan emits.Describe the solution you'd like
In
supports_filters_pushdown, returnTableProviderFilterPushDown::Exactfor filters thatconvert_filter_to_predicatetranslates losslessly into an icebergPredicate, andInexact(orUnsupported) otherwise.The non-trivial part is detecting lossy conversions.
Willingness to contribute
I would be willing to contribute to this feature with guidance from the Iceberg Rust community