feat(index): merge incremental inverted index segments

[BubbleCal]

Fixes #6292

Feature

What is the new feature?

Inverted / FTS index optimization now honors num_indices_to_merge for incremental index segments. num_indices_to_merge=0 keeps append-only delta behavior, while num_indices_to_merge=N folds newly indexed fragments together with the latest N existing inverted segments into one replacement segment.

Why do we need this feature?

Repeated incremental optimize_indices() calls could accumulate many FTS index segments even though the API already exposed num_indices_to_merge. That made the public optimize contract inconsistent for inverted indexes and could leave unnecessary segments for query-time search.

How does it work?

The scalar optimize path now has an inverted-index-specific branch. It selects the requested existing FTS segments, loads them copy-on-write, filters stale / deleted old rows, merges them with newly tokenized data, writes one new inverted index root, and removes exactly the selected old metadata segments. The implementation preserves tokenizer parameters, positions settings, token-set format, FTS format version, posting-tail codec, deleted_fragments, and stable-row-id filtering semantics.

Validation

• cargo fmt --all
• cargo clippy -p lance-index --all-targets -- -D warnings
• cargo clippy -p lance --all-targets -- -D warnings
• cargo test -p lance test_optimize_fts_respects_num_indices_to_merge
• cargo test -p lance test_optimize_fts
• cargo test -p lance test_fts_index_incremental_reindex_after_in_place_update
• cargo test -p lance cleanup_lineage
• cargo test -p lance auto_clean_referenced_branches
• uv run pytest python/tests/test_scalar_index.py::test_fts_optimize_num_indices_to_merge

[codecov]

Codecov Report

❌ Patch coverage is 84.48000% with 97 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/append.rs	64.84%	36 Missing and 9 partials ⚠️
rust/lance-index/src/scalar/inverted/index.rs	82.62%	23 Missing and 14 partials ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	87.50%	1 Missing and 14 partials ⚠️

📢 Thoughts on this report? Let us know!

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

_{Tip: disable this comment in your organization's Code Review settings.}