fix(index): skip unrecognized index types

[lance-gatefixer]

Summary

• resolve index readers by exact, case-insensitive protobuf identity, including released 0.36 scalar aliases
• expose only indices this build can read while preserving opaque metadata in the complete manifest view
• apply the same reader/version boundary to query selection, optimization, commit-time migration, and compaction
• cover unknown-type fallback, preservation, migration, and unremappable compaction behavior

Root cause

Unknown index detail types were retained in the manifest but still entered the usable-index view and maintenance paths. Query selection, optimization, commit-time migration, and compaction could therefore try to open opaque metadata, fail unrelated work, or rewrite fragments without preserving the foreign index coverage.

Validation

• cargo test -p lance --lib test_unknown_index_type_does_not_block_queries_or_optimization
• cargo test -p lance --lib test_remapper_not_created_for_unknown_index_type
• cargo test -p lance --lib test_unsupported_index_without_a_bitmap_does_not_fail_later_commits
• cargo test -p lance --lib test_compaction_defers_fragments_an_unsupported_index_covers
• cargo test -p lance-index test_supports_details_matches_complete_type_name_case_insensitively
• cargo test -p lance --lib test_v036_scalar_details_are_still_known
• cargo test -p lance --lib test_remapper_not_created_without_remappable_indices
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

Fixes #8528

[lance-gatefixer]

Blocked: The current-head Rust linux-arm check is failing from an inherited DataFusion memory-pool exhaustion outside this repair diff.

At remote head c86443bd668cb5b2b2eeec6a162fbd7b2d255034, the linux-arm job 96953077588 failed its test step when an external-sort merge requested 40.0 MB with only 30.0 MB available in a shared 150.0 MB pool. Current main tip 55f5ac5cdd778df673598bd58e531b54b74e1ec3 is an ancestor of this head. The failing JSON BTree test and allocation path are unchanged from main, and the exact test passes under the locked CI profile with every lance-index feature, so the evidence does not support an attributable in-scope code change.

Re-run the failed linux-arm job as the smallest next action. If it reproduces, address the shared DataFusion test memory or concurrency configuration in a dedicated fix; reducing the test runtime sort-spill reservation is a practical alternative.

[lance-gatefixer]

Blocked: The current-head Python Linux 3.10 x86_64 and Rust windows-build checks are failing in two unrelated flaky tests that are unchanged from main, so there is no evidence-backed repair change to make on this branch.

At remote head 47fd70368f2619a5876160a21ae1d677bb3460af, Python job 99038205736 fails test_torch_index_with_nans[V3] with StopIteration while sampling residuals, and Rust Windows job 99037476495 fails test_paginated_lister_throttle_errors_decrease_rate because current_rate() < 100.0 was false. Current main tip 108f78e927266d8a8ee14dadc74d455f7242dd85 is an ancestor of this head, and python/tests/test_vector_index.py plus rust/lance-io/src/object_store/throttle.rs are unchanged from main. That same base tip passed the corresponding Python 3.10 job 99034088741 and Windows job 99033341399; the focused throttle test also passes locally.

Re-run the two failed jobs as the smallest next action. If either failure recurs, stabilize the NaN sampling and throttle timing tests in dedicated fixes as the practical alternative.

[lance-gatefixer]

Blocked: Three current-head checks remain failed for non-attributable flaky or runner-resource conditions, so no evidence-backed repair change can clear CI.

At remote head 542e4d2d69f5d8458df292763ea084ce42801e9b, linux-arm job 99040953655 missed the unchanged randomized HNSW/PQ recall threshold by five rows (123/256), Python 3.10 job 99041508413 repeats the unchanged test_torch_index_with_nans[V3] sampling StopIteration, and linux-build job 99040953616 could not allocate either io_uring worker thread. Current main tip 108f78e927266d8a8ee14dadc74d455f7242dd85 is an ancestor of this head; its corresponding Python and ARM jobs passed, while its Linux job failed the same io_uring allocation path. The exact HNSW/PQ case passes locally under the locked CI profile, and none of the failing test or implementation files differs from main.

Re-run the three failed jobs as the smallest next action. If they recur, stabilize the NaN sampling and approximate-recall fixtures and address Linux runner memory/io_uring capacity in dedicated fixes.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The feature patch is unchanged after the base refresh. The new column-stitching compaction path remains behind the same unreadable-index planning and commit boundaries, and the focused query, migration, and compaction regressions pass on this head.