perf(index): fast-path total rows and cache index_statistics output

[justinrmiller]

Summary

Two optimizations to Dataset::index_statistics that together turn a ~5 ms call into ~22 µs for the common repeat-call pattern on a 10 M-row / 10 K-fragment dataset, with exact-behavior fallback for legacy datasets.

P1 — skip count_rows(None) when the manifest can answer

gather_fragment_statistics was calling Dataset::count_rows(None) purely to derive num_unindexed_rows = total - indexed. On modern datasets that's an in-memory fan-out; on legacy fragments without physical_rows / writer_version it opens the first data file of every fragment from object storage. New manifest_total_rows helper sums Fragment::num_rows() in memory and falls through to the existing count_rows path only when any fragment is missing that metadata.

P2 — cache the JSON keyed on manifest version

index_statistics is a pure function of (dataset URI, manifest version, index name). The DSIndexCache already scopes entries by dataset URI; manifest version bumps on every operation that can change the answer (append, delete, compact, index create/optimize/drop), so keying on it gives automatic invalidation with no write-path coupling. New IndexStatisticsKey alongside the existing IndexMetadataKey / ScalarIndexDetailsKey; index_statistics wraps the existing dispatch with a get/insert.

Errors don't poison the cache. No manifest format change, no public API change.

Benchmarks

New benches/index_stats.rs. Runs four paths in a single process on a shared fixture:

• count_rows_baseline — cost of the count_rows(None) fan-out alone.
• legacy_cold — pre-P1/P2 behavior, exercised via #[doc(hidden)] bench_legacy_index_statistics.
• cold — current behavior, fresh session per iter.
• cached — current behavior, warm cache.

A startup parity check asserts the legacy and current paths produce identical counters (num_indexed_rows, num_unindexed_rows, num_indexed_fragments, num_unindexed_fragments, num_indices) before measurement begins.

BTree index on Int32, local SSD, Apple Silicon, --sample-size 10 --measurement-time 2s.

Scaling across fixture size

Fixture	count_rows_baseline	legacy_cold	cold (P1)	cached (P1 + P2)
256 K rows / 256 frags	58 µs	538 µs	466 µs	1.4 µs
1 M rows / 1 024 frags	190 µs	834 µs	643 µs	2.9 µs
10 M rows / 10 000 frags	1.82 ms	5.33 ms	3.56 ms (−33%)	21.7 µs (~245×)

Observations:

• count_rows cost scales linearly with fragment count (~180–200 ns/fragment in-memory). On legacy-format fragments or slow object stores it grows by orders of magnitude — each fragment may require a HEAD plus a range read to recover physical_rows.
• P1's absolute savings scale with fragment count (72 µs → 191 µs → 1 770 µs). The cold path converges toward legacy_cold − count_rows_baseline, which is the theoretical floor.
• P2's cache-hit cost is roughly constant in fixture size (dominated by the moka lookup + string clone, not JSON size).
• Together, P1 + P2 convert a 5 ms call into a 22 µs call at 10 K fragments.

Test plan

• cargo test -p lance --lib -- index::tests:: — 68 passing, including 5 new:
  • test_index_statistics_row_counts_match_count_rows (multi-fragment + partial index + deletes + append; asserts indexed + unindexed == count_rows(None))
  • test_manifest_total_rows_missing_metadata_returns_none (forces the count_rows fallback)
  • test_index_statistics_cache_hit_avoids_io (second call does zero reads per io_tracker)
  • test_index_statistics_cache_invalidates_on_manifest_bump (append + optimize_indices both return fresh numbers)
  • test_index_statistics_cache_distinguishes_index_names (two indices on the same dataset don't collide)
• cargo clippy --all --tests --benches -- -D warnings
• cargo fmt --all -- --check
• cargo bench -p lance --bench index_stats — parity check passes, numbers above.

Not in this change

Left for follow-up, in rough priority order:

• Single-pass indexed_fragments (currently O(deltas × fragments) with Fragment clones). Cheap rewrite, ~5–10× on many-delta indices.
• load_statistics for BTree / Inverted / BloomFilter / LabelList — only Bitmap has it today; others fall back to open_generic_index, which materializes the full index file. 20–50× potential on cold-cache calls, medium-effort per-plugin work.
• Persisted num_indexed_rows on IndexMetadata — manifest format change; probably unnecessary after P1 + P2.
• Related production signal: ENT-547 / index_statistics on LABEL_LIST index is very slow #4620 flagged slow index_stats; test(java): re-construct tests for merge operation #4483 added a Phalanx-side timeout as mitigation.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 81.60000% with 46 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index.rs	81.14%	45 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!