Skip to content

feat: add inexact not filter pushdown#488

Open
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:feat/not-inexact-pushdown
Open

feat: add inexact not filter pushdown#488
liujiwen-up wants to merge 1 commit into
apache:mainfrom
liujiwen-up:feat/not-inexact-pushdown

Conversation

@liujiwen-up

@liujiwen-up liujiwen-up commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: none.

This PR adds conservative SQL NOT pushdown after the exact residual reader semantics introduced around filter pushdown. SQL NOT cannot be treated as exact because Paimon's predicate evaluation uses two-valued predicate semantics while DataFusion must preserve SQL three-valued logic, especially for NULL values.

Brief change log

  • Push translatable SQL NOT(expr) predicates to Paimon as Predicate::Not, while marking the DataFusion filter pushdown result as Inexact so the residual filter is preserved.
  • Push case-sensitive NOT LIKE as an inexact negated LIKE predicate when the positive LIKE pattern can be translated; ILIKE remains unsupported.
  • Track whether translated predicates require a residual filter, so partition-only NOT is not incorrectly classified as exact.
  • Skip limit pushdown when the pushed filter requires a residual filter.
  • Add conservative stats pruning for Predicate::Not(inner): prune only when the inner predicate must match every row in the file or data-evolution group; otherwise fail open.

Tests

  • cargo fmt -- --check: passed
  • git diff --check: passed
  • PYO3_PYTHON=/opt/homebrew/bin/python3.12 cargo clippy --all-targets --workspace --features fulltext,vortex,mosaic -- -D warnings: passed
  • cargo test -p paimon-datafusion filter_pushdown --lib: passed, 36 tests
  • cargo test -p paimon-datafusion table::tests::test_scan_pushes_not_as_inexact_and_skips_limit_hint --lib: passed
  • cargo test -p paimon-datafusion table::tests::test_scan_partially_translated_not_filter_prunes_partitions_but_skips_limit_hint --lib -- --exact: passed after make docker-up
  • cargo test -p paimon-datafusion --test sql_context_tests test_not_filter_pushdown_keeps_sql_null_semantics: passed
  • cargo test -p paimon predicate_stats --lib: passed, 19 tests
  • cargo test -p paimon table_scan::tests::test_data_file_matches_not --lib: passed, 3 tests
  • cargo test -p paimon table_scan::tests::test_data_evolution_group_matches_not --lib: passed
  • cargo test -p paimon-datafusion --all-targets: passed through the previously failing test; local run then failed only on vector_search_top3 and vector_search_top6_returns_all because liblumina_py.dylib is not installed locally.

API and Format

No public API, manifest, snapshot, or storage format changes are introduced.

Documentation

No user-facing documentation changes are required. The behavior is covered by DataFusion pushdown and core stats unit tests.

@liujiwen-up liujiwen-up changed the title feat: add inexact NOT filter pushdown feat: add inexact not filter pushdown Jul 8, 2026
@liujiwen-up liujiwen-up force-pushed the feat/not-inexact-pushdown branch from 153c8dc to 2c34c50 Compare July 8, 2026 13:51
@liujiwen-up liujiwen-up force-pushed the feat/not-inexact-pushdown branch from 2c34c50 to acd2734 Compare July 8, 2026 14:19

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants