vector-search: support scalar pre-filter before Top-K

[shyjsarah]

Summary

Support scalar pre-filtering before Top-K for data-evolution/global-index vector search, including DataFusion literal and lateral vector_search queries.

The scalar predicate is evaluated against the same pinned snapshot as vector search, producing global row IDs that are localized per vector-index shard and passed to the vector backend as an allow-list. A normal Paimon read is used so scalar global indexes such as BTree can prune the filter read while residual evaluation, partial index coverage, and deletion visibility remain exact.

Changes

• Enable VectorSearchBuilder::with_filter and BatchVectorSearchBuilder::with_filter on the data-evolution path.
• Pin scalar filtering, deletion-vector handling, raw fallback, refine, and materialization to one resolved snapshot.
• Apply the same global row-ID allow-list to indexed search and unindexed raw fallback.
• Localize a shared bitmap once per vector-index shard, skip empty shards, and share bitmaps across batch queries without deep copies.
• Add a reusable prepared-filter API so lateral execution evaluates a target-side scalar predicate once per physical plan.
• Push fully translated DataFusion predicates into literal vector_search while retaining an Inexact residual correctness check.
• Push fully translated target-only conjuncts into lateral vector search while preserving left-side, cross-side, unsupported, and inexact residual predicates.
• Reject NaN predicates from vector pre-filter pushdown because Paimon and Arrow comparison semantics differ.
• Show pushed scalar filters in logical and physical EXPLAIN output.
• Document scalar pre-filter semantics and lateral-query behavior.

Testing

• cargo fmt --all --check
• git diff --check
• cargo test -p paimon --lib table::vector_search_builder::tests
• cargo test -p paimon-datafusion --lib filter_pushdown::tests::test_vector_prefilter_rejects_nan_literals
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vindex_build_then_vector_search_query -- --nocapture
• cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture
• cargo clippy -p paimon --lib --tests -- -D warnings
• cargo clippy -p paimon-datafusion --lib --tests -- -D warnings

Covered cases include filter-before-Top-K ordering, empty matches, batch reuse, non-zero shard offsets, raw fallback for unindexed rows, time-travel snapshot pinning, literal SQL, lateral SQL, mixed/cross-side conjuncts, and NaN pushdown safety.

Notes

• No on-disk format change.
• No SQL function signature change.
• Existing VectorSearch::include_row_ids public API remains unchanged; this PR adds PreparedVectorSearchFilter and shared-filter builder support.
• This is scalar pre-filtering for vector search, not the existing vector/full-text route-fusion hybrid_search feature.

[JingsongLi]

+1