feat: Support Distributed Segmented BTREE Index Building

[beinan]

Summary

This PR implements Segmented BTREE Index support when building distributed scalar indexes with daft-lance using the new segmented=True parameter.

What's Changing

• Introduced SegmentedFragmentIndexHandler that builds a fully formed independent lance.Index segment on each worker.
• Replaced the manual index transaction creation block on the coordinator with a clean, native atomic commit of the gathered index segments using commit_existing_index_segments().
• Added extensive test coverage for segmented BTree indexes (string/integer columns, multiple segments, replacements, and fallback behaviors).
• Upgraded the pylance dependency constraint to >=7.0.0 to support the required segment commitment APIs.

Why This is Better

1. Resolves describe_indices() Crash: The previous manual transaction building block produced indices with empty index_details. Downstream tooling calling describe_indices() would crash. This change preserves proper protobuf-serialized index_details in the committed index segments.
2. Clean Transactions: Uses Lance's native atomic index commitment mechanics instead of low-level LanceOperation crafting.

---

🚀 Architectural Benefits for Extremely Large Tables

Building indices on extremely large tables (multi-billion rows, hundreds of gigabytes/terabytes of data) introduces severe scaling and performance bottlenecks. Adopting a segmented index building approach over a single monolithic file provides immense advantages:

1. Zero Shuffling or Global Sorting Overhead

A BTree index is fundamentally a sorted data structure. Under the classic partitioned-and-merged approach, merging parallel partitions requires a global K-way sort merge across workers' outputs.

• The bottleneck: For multi-terabyte datasets, doing a sort-merge over S3/network page storage creates massive memory pressure and I/O bottlenecks on the coordinator.
• Segmented benefit: Each worker builds a fully formed, independent, pre-sorted index segment for its local fragments. There is zero coordination, zero shuffling, and zero global sorting required at the end. The merge step is reduced to an $O(1)$ atomic metadata transaction registering the existing segments.

2. Elastic Horizontal Scalability (No Coordinator Bottlenecks)

In the partitioned-and-merged model, the merge phase runs entirely on the coordinator node.

• The bottleneck: The coordinator becomes a hard bottleneck; as dataset size and worker counts scale up, the coordinator's CPU and memory requirements grow quadratically, often resulting in Out-Of-Memory (OOM) crashes.
• Segmented benefit: All computation scales horizontally with your Daft workers. The coordinator only executes a lightweight transaction commit registering pre-built segments. You can scale to thousands of workers without bottlenecking the coordinator.

3. Highly Cost-Effective Incremental index Updates (Append-Friendly)

In real-world data lakes, datasets are rarely static; new data is appended constantly.

• The bottleneck: With a monolithic index, appending new fragments forces a full rewrite of the entire unified index. Re-indexing a 10TB dataset to index a new 50GB append is prohibitively expensive.
• Segmented benefit: Since the index is segmented, you can build new index segments for only the newly appended fragments, and atomically attach them to the existing index using commit_existing_index_segments. Already-indexed data remains untouched, turning index maintenance costs from $O(\text{Total Dataset})$ to $O(\text{Append Size})$.

4. Resiliency & Fault-Tolerance

If a distributed job fails mid-way:

• Legacy: All work is lost because a single unified file must be created at the end.
• Segmented: Individual worker tasks commit fully formed index segments. Failed worker tasks can be retried independently without invalidating or discarding the successfully built segments of other workers.

---

Test Plan

Ran pytest on local suite:

• Added comprehensive unit tests in test_lancedb_scalar_index.py verifying multiple segment generation, query correctness, index description, and fallback workflows.
• All 29 tests pass successfully with no regressions.

🤖 Generated with Claude Code

[rchowell]

Thanks for the contribution!