feat: add segmented BTree index planning and merge support

[zhangyue19921010]

Closes: #6979

PAY ATTENTION

Refactored scalar index optimize logic in append.rs:

1. Extracted merge_scalar_indices unified function — consolidates all scalar index merge logic from merge_indices_with_unindexed_frags into a reusable function with explicit segment selection.

2. Added BTreeIndex::merge_segments — k-way segment merge for BTree, enabling multi-segment consolidation without re-reading the dataset.

3. BTree append mode now creates delta segments — num_indices_to_merge=0 builds an index only for unindexed fragments, preserving existing segments (reduces write amplification).

4. Non-BTree types (like Bitmap/LabelList) behavior preserved — On main, these types always maintain a single segment: every optimize call (regardless of num_indices_to_merge) merges old index + new data into one segment via update(). This PR preserves that exact behavior. Additionally, a safety guard rejects multi-segment state for these types with a clear error, since they lack merge N primitives and should never accumulate multiple segments.

Compatibility Testing

Context

Main branch ignores num_indices_to_merge for scalar indices — all modes behave identically (always merge the single old segment + new data → 1 segment). Multi-segment scalar index is a state that cannot exist on main.

Test Results (single-segment starting state, which is the only state main can produce)

Index Type	Mode	Main	PR	Diff
BTree	default	1 seg (param ignored)	1 seg (merge all)	✅ Same result
BTree	merge_1	1 seg (param ignored)	1 seg (merge 1 old + new)	✅ Same result
BTree	merge_2	1 seg (param ignored)	1 seg (merge 2 old, but only 1 exists)	✅ Same result
BTree	append	1 seg (param ignored)	2 seg (delta only)	⚠️ Changed
Bitmap	all modes	1 seg	1 seg	✅ Same
LabelList	all modes	1 seg	1 seg	✅ Same

PR-only new behavior: multi-segment BTree scenarios (unreachable on main)

After accumulating N segments via repeated append-mode optimizes:

Starting state	Mode	Result
3 segments + new data	default	3−3+1 = 1 segment
3 segments + new data	merge_1	3−1+1 = 3 segments
3 segments + new data	merge_2	3−2+1 = 2 segments
3 segments + new data	append	3+1 = 4 segments

Query correctness: All combinations verified — no data loss or incorrect results.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 87.68719% with 148 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/scalar/btree.rs	65.25%	63 Missing and 27 partials ⚠️
rust/lance/src/index/append.rs	93.56%	15 Missing and 14 partials ⚠️
rust/lance-index/src/scalar/btree.rs	63.63%	18 Missing and 6 partials ⚠️
rust/lance/src/index/create.rs	98.80%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!