feat: add inexact not filter pushdown#488
Open
liujiwen-up wants to merge 1 commit into
Open
Conversation
153c8dc to
2c34c50
Compare
2c34c50 to
acd2734
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: none.
This PR adds conservative SQL
NOTpushdown after the exact residual reader semantics introduced around filter pushdown. SQLNOTcannot be treated as exact because Paimon's predicate evaluation uses two-valued predicate semantics while DataFusion must preserve SQL three-valued logic, especially for NULL values.Brief change log
NOT(expr)predicates to Paimon asPredicate::Not, while marking the DataFusion filter pushdown result asInexactso the residual filter is preserved.NOT LIKEas an inexact negated LIKE predicate when the positive LIKE pattern can be translated;ILIKEremains unsupported.NOTis not incorrectly classified as exact.Predicate::Not(inner): prune only when the inner predicate must match every row in the file or data-evolution group; otherwise fail open.Tests
cargo fmt -- --check: passedgit diff --check: passedPYO3_PYTHON=/opt/homebrew/bin/python3.12 cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings: passedcargo test -p paimon-datafusion filter_pushdown --lib: passed, 36 testscargo test -p paimon-datafusion table::tests::test_scan_pushes_not_as_inexact_and_skips_limit_hint --lib: passedcargo test -p paimon-datafusion table::tests::test_scan_partially_translated_not_filter_prunes_partitions_but_skips_limit_hint --lib -- --exact: passed aftermake docker-upcargo test -p paimon-datafusion --test sql_context_tests test_not_filter_pushdown_keeps_sql_null_semantics: passedcargo test -p paimon predicate_stats --lib: passed, 19 testscargo test -p paimon table_scan::tests::test_data_file_matches_not --lib: passed, 3 testscargo test -p paimon table_scan::tests::test_data_evolution_group_matches_not --lib: passedcargo test -p paimon-datafusion --all-targets: passed through the previously failing test; local run then failed only onvector_search_top3andvector_search_top6_returns_allbecauseliblumina_py.dylibis not installed locally.API and Format
No public API, manifest, snapshot, or storage format changes are introduced.
Documentation
No user-facing documentation changes are required. The behavior is covered by DataFusion pushdown and core stats unit tests.