perf(scalar): cache store-free state for the zone map index

[Ali2Arslan]

Summary

Opening a zone map index cached the live Arc<dyn ScalarIndex>. A ZoneMapIndex holds an IndexStore, which transitively pins the session-scoped object store, metadata cache, and scan scheduler. So every cached zone map index:

• retained those shared, session-scoped resources for the lifetime of the cache entry, and
• mis-attributed them under deep_size_of — each opened index reported the whole shared cache as its own bytes, N-times double-counting it.

This is the general problem raised in the #7574 review discussion: a cache entry should never hold a session-scoped resource.

This PR applies the fix to the zone map index by following the pattern already used by BitmapIndex / BTreeIndex / LabelListIndex:

• Add ZoneMapIndexState — a serializable, store-free snapshot (the zone-statistics RecordBatch + rows_per_zone), persisted through a CacheCodec.
• Cache only that state; inject the IndexStore, cache handle, and fragment-reuse index at reconstruct time. ZoneMapIndex::try_from_serialized already rebuilds the index with no IO, so reconstruct is free.
• Override the plugin's get_from_cache / put_in_cache / get_or_insert_in_cache to route through the state via single_flight_open (coalesces concurrent cold opens).
• Extract the existing zones -> RecordBatch encoding into a shared zones_to_batch helper (used by both the on-disk writer and the cache codec), replacing the builder's inline copy.

Behavior is unchanged; only the cached representation changes.

Follow-ups

The remaining scalar indices that still cache a live object holding an IndexStore — ngram, rtree, inverted, json — can migrate onto this same template in separate, focused PRs.

Test plan

• test_zonemap_state_codec_roundtrip — CacheCodec serialize/deserialize is lossless and matches a fresh encode of the live index.
• test_zonemap_state_reconstruct_is_io_free — reconstruct against a store rooted at an empty dir (missing index file) still searches identically, proving no store IO on reconstruct/search.
• test_zonemap_state_deep_size_excludes_shared_resources — the cached state's deep_size_of counts only its batch (guards against re-adding a shared field).
• test_zonemap_plugin_cache_is_single_flight_state — end-to-end through the plugin: a ZoneMapIndexState (not a live index) is cached, cold/warm opens coalesce onto a single load, and both opens search identically.
• cargo test -p lance-index --lib scalar::zonemap (22 passed) and full scalar:: suite (395 passed).
• cargo fmt --all + cargo clippy -p lance-index --tests -- -D warnings clean.

Made with Cursor

[codecov]

Codecov Report

❌ Patch coverage is 88.20961% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/zonemap.rs	88.20%	13 Missing and 14 partials ⚠️

📢 Thoughts on this report? Let us know!