Skip to content

[WIP][core] Support deletion vectors in global index searches#8451

Closed
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/dv-global-index-search
Closed

[WIP][core] Support deletion vectors in global index searches#8451
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/dv-global-index-search

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR makes global index based reads aware of deletion vectors so deleted row IDs are not returned by shard global index scans, vector search, or full-text search. It also preserves deletion file metadata while planning global-index build splits so index builders read only live rows.

Changes

  • Add a shared deletion-vector row ID filter that reads snapshot data splits, converts candidate row ranges and deletion vectors to RoaringNavigableMap64, and filters with bitmap operations.
  • Apply deletion-vector filtering to GlobalIndexScanner results, including unindexed row ranges.
  • Apply live-row filters before vector index topK search and add a final safety filter after reranking for single and batch vector search.
  • Apply deletion-vector filtering before full-text final topK, including compound query and raw fallback paths.
  • Preserve aligned deletion files when creating shard indexed splits and when sorted global index build splits are regrouped by row range.
  • Update Spark global-index topology planning to use the snapshot-aware split builder.

Testing

  • git diff --check
  • mvn -pl paimon-core,paimon-spark/paimon-spark-common -am -DskipTests compile
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=DeletionVectorTest#testBitmapDeletionVectorTo64,VectorSearchBuilderTest#testVectorSearchExcludesDeletionVectorRowsBeforeTopK+testGlobalIndexScannerExcludesDeletionVectorRows+testShardIndexedSplitsPreserveDeletionFiles,FullTextSearchBuilderTest#testFullTextSearchExcludesDeletionVectorRowsBeforeTopK,SortedGlobalIndexBuilderSplitTest#testSplitByContiguousRowRangePreservesDeletionFiles test

Notes

No migration is required. The existing overloads without a snapshot keep their previous behavior; snapshot-aware planning is used by the Spark global index build path.

@JingsongLi JingsongLi force-pushed the codex/dv-global-index-search branch from c27d849 to 7939b2a Compare July 3, 2026 15:25
@JingsongLi JingsongLi changed the title [core] Support deletion vectors in global index searches [WIP][core] Support deletion vectors in global index searches Jul 3, 2026
@JingsongLi JingsongLi force-pushed the codex/dv-global-index-search branch from 7939b2a to df1b828 Compare July 3, 2026 15:31
@JingsongLi JingsongLi marked this pull request as draft July 4, 2026 04:29
@JingsongLi JingsongLi closed this Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant