Stale vector index after compaction: fragment_bitmap inconsistency causes query failure

[sinianluoye]

Summary

After compaction, vector index queries can fail with:

specified fragment id 465 does not exist in the dataset

The root cause is that DatasetIndexRemapper::remap_indices() skips remapping for indices whose fragment_bitmap doesn't intersect with the affected fragments. When the bitmap is stale (contains deleted fragment IDs), the remap is incorrectly skipped, leaving stale ROW_IDs in the index.

Key Code Locations

• rust/lance/src/dataset/index.rs:66-72 — remap_indices() bitmap intersection check: skips remap when bitmap doesn't intersect affected fragments
• rust/lance/src/dataset/transaction.rs:2456-2488 — recalculate_fragment_bitmap: recalculates bitmap but doesn't validate against actual dataset fragments
• rust/lance/src/index/append.rs:273-277 — Vector index merge path uses raw fragment_bitmap (the scalar index path at lines 162-168 correctly intersects with dataset.fragment_bitmap)

Root Cause Analysis

The vector index optimize_indices merge path (append.rs:273-277) uses the original fragment_bitmap rather than the effective_fragment_bitmap. This is inconsistent with the scalar index path, which intersects with dataset.fragment_bitmap to filter out deleted fragments.

This causes deleted fragment IDs to persist in the bitmap. When a subsequent compaction runs, remap_indices() checks the (stale) bitmap and incorrectly determines that no remap is needed for certain indices, leaving stale ROW_IDs that reference deleted fragments.

Reproduction

Repeatedly call optimize(OptimizeAction::All) (which runs both compact and optimize_indices). Eventually the index will contain ROW_IDs pointing to fragments that have been compacted away.

Suggested Fix Directions

1. Vector merge path: Use effective_fragment_bitmap instead of raw fragment_bitmap in the vector index merge path (consistent with the scalar path)
2. Relax bitmap check in remap_indices(): Remove or weaken the bitmap intersection check so that indices are always remapped when compaction affects any fragments
3. Safety check in recalculate_fragment_bitmap: Validate the recalculated bitmap against the actual dataset fragment list, removing any fragment IDs that no longer exist