fix(index): exclude shared metadata cache from LanceIndexStore deep size

[Ali2Arslan]

Summary

LanceIndexStore::deep_size_of_children counted its metadata_cache:

self.object_store.deep_size_of_children(context)
    + self.index_dir.as_ref().deep_size_of_children(context)
    + self.metadata_cache.deep_size_of_children(context)   // <-- shared, session-scoped

But that cache is session-scoped and shared. Session::deep_size_of_children already accounts for it once (rust/lance/src/session.rs), and LanceCache::deep_size_of_children returns approx_size_bytes() while ignoring the dedup Context. So the shared cache isn't deduplicated: every opened index that holds a store re-counts the entire cache as its own bytes, inflating and N-times double-counting reported memory (N = number of open index stores).

Change

Count only the store's own footprint — the object store and index dir. The shared metadata cache is left to Session to report once.

Test plan

• test_store_deep_size_excludes_metadata_cache — inserting a 4 MB blob into the shared cache must not change store.deep_size_of(). Verified it fails before the fix (before: 276, after: 4,194,620) and passes after.
• cargo fmt --all and cargo clippy -p lance-index --all-targets -- -D warnings clean.

Made with Cursor

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[wjones127]

issue(non-blocking): Removing metadata_cache seems correct, but object store is another shared object here. I wonder if the real fix is making LanceIndexStore not part of the cache entries themselves, instead just making it cheap to reconstruct LanceIndexStore on load. You can see how we did this for Vector indices with IvfStateEntry:

lance/rust/lance/src/index/vector/ivf/v2.rs

Lines 228 to 238 in 71c4aa2

	pub(crate) trait IvfStateEntry: DeepSizeOf + Send + Sync + 'static {
	fn serialize_state(&self, w: &mut CacheEntryWriter<'_>) -> Result<()>;
	fn reconstruct<'a>(
	&'a self,
	object_store: Arc<ObjectStore>,
	file_metadata_cache: &'a LanceCache,
	index_cache: LanceCache,
	frag_reuse_index: Option<Arc<FragReuseIndex>>,
	) -> BoxFuture<'a, Result<Arc<dyn VectorIndex>>>;
	}

[Ali2Arslan]

Thanks Will! I'll take a look and follow up

[Ali2Arslan]

hmm, I took a look, so you're saying we should remove LanceIndexStore from all the scalar indexes so that we're not caching it and instead it's always reconstructed? Similar to what some are already doing with their *State structs?

[Ali2Arslan]

I can enumerate the scalar indexes which don't currently have a *State + reconstruct impls

[Ali2Arslan]

Is this general direction good with you Will? #7652
If so, I can create PR's for the other indexes as well and then drop the LanceIndexStore.