Feature: Add Bloom Filter Support for Iceberg read/write operations

[navneethk-xch]

Summary

DAFT recently migrated to arrow-rs, which supports Parquet bloom filters. Iceberg already exposes bloom filter configuration via table properties. This proposal enables DAFT to pass through and honor these properties during write_iceberg() and leverage them implicitly during read_iceberg() without introducing new user-facing APIs.

Currently iceberg spec supports the following configurations for adding bloom filters to table properties:

Motivation

• Bloom filters can significantly improve file pruning for equality-based predicates on high-cardinality columns (e.g., user_id, email, trace_id).
• With arrow-rs support in place, DAFT can enable this with minimal changes while keeping the API surface same.

Example Usage (Pseudo-code)

Table properties [configured with bloom filters]

properties={
        "write.parquet.bloom-filter-enabled.column.user_id": "true",
        "write.parquet.bloom-filter-fpp.column.user_id": "0.01",
        "write.parquet.bloom-filter-enabled.column.email": "true",
        "write.parquet.bloom-filter-fpp.column.email": "0.001",
        "write.parquet.bloom-filter-max-bytes": "1048576"
)

Filtering on the column when reading from Iceberg

df = df.filter(daft.col("user_id") == "abc123")

Bloom filters should be leveraged automatically during scan where applicable.

Corner Cases [Some cases to consider]

1. Table properties not picked up by writer

2. Partial configuration (enabled but missing params)

write.parquet.bloom-filter-enabled.column.user_id = true
# no fpp provided

3. Schema Evolution

Old files → no bloom filters
New files → with bloom filters

4. Missing bloom filters at read time

5. Unsupported column types at write time

6. Write modes (append / overwrite)