feat: support IVF partitions multi-split

[BubbleCal]

Feature

What is the new feature?

This PR allows a single optimize_indices call on the v3 IVF incremental optimize path to split multiple oversized IVF partitions in one pass.

Why do we need this feature?

Previously, optimize could split at most one oversized partition per run. After large appends, several partitions can exceed the split threshold at the same time, which forced repeated optimize cycles to bring the index back into a healthy partition layout.

How does it work?

• check_partition_adjustment now collects all split candidates from the current snapshot and keeps the existing single-partition join fallback.
• The multi-split path preserves existing partition ids and appends one new partition per split partition.
• When any split happens, optimize continues to merge all existing delta indices in the same round, preserving the existing merge semantics.
• Candidate rows from overlapping reassign partitions are resolved globally so the same row is moved at most once, choosing the best destination by distance.

Performance Improvement

What is the performance issue or bottleneck?

The initial multi-split implementation removed the functional limitation, but overlapping reassign partitions still had avoidable overhead:

• split planning was done sequentially
• split plans retained full raw vector payloads longer than necessary
• a reused candidate partition recomputed its baseline distance to the original centroid for every overlapping split request

How does this PR improve performance?

• split plans are now built with bounded parallelism across compute CPUs
• split plans no longer retain raw partition vectors after producing the original-partition assign ops
• each reused candidate partition is loaded once and computes its baseline distance once, then reuses that result across overlapping split requests
• best candidate moves are updated in-place per row id instead of materializing intermediate move vectors per request

Testing

• cargo test -p lance compute_reassign_candidate_moves_vectors_to_new_centroids
• cargo test -p lance test_partition_split_on_append_multivec
• cargo test -p lance test_split_multiple_partitions_in_one_optimize
• cargo test -p lance test_join_partition_on_delete_multivec

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[codecov]

Codecov Report

❌ Patch coverage is 90.73634% with 39 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/builder.rs	87.62%	22 Missing and 15 partials ⚠️
rust/lance/src/index/vector/ivf/v2.rs	98.36%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

load_partition_raw_vectors will load multivector into flat ones so one row id could match different vectors. Will this make the multivector data been missing in our index?

[Xuanwo]

The centroid2_part_idx is calculated using ivf.num_partitions(). Could it be that some partitions have been filtered, causing the centroid2_part_idx in the plan to exceed the actual number? If so, the subsequent assign_ops[*target_idx] operation might panic.

[Xuanwo]

Thank you for working on this!