Lazy roaring flags bitmap and bool index counts

[generall]

What

Opening a read-only segment scanned every flags file end to end. ReadOnlyRoaringFlags::open materialized the whole RoaringBitmap via iter_ones(), and since every payload field carries a null index, that cost was paid per field per segment — for bitmaps most queries never touch. It defeats the point of prefetching only the bytes a query needs.

Note the counts were not the culprit: ReadOnlyNullIndex::open computes none at all (indexed_points_count() is has_values_flags().len(), straight from the tiny status file). The eager scan was the bitmap itself, so that is where the laziness had to go.

• ReadOnlyRoaringFlags::bitmap is now a OnceLock<RoaringBitmap>, filled by a scan on first access. open touches only the status file. (OnceLock because indexes are queried through &self across threads; get_or_try_init is still unstable, so a race may build twice and drop the loser.)
• ReadOnlyBoolIndex's three eager count fields collapse into one lazily-derived, cached BoolCounts. Its live_reload refreshes them in place when they are already present, and leaves them unset otherwise — so reloading an index nothing queries stays scan-free. The refresh is free when it runs: counts can only exist if deriving them already materialized both bitmaps, which live_reload patches in place.
• ram_usage_bytes stays infallible: an unmaterialized bitmap holds no RAM, so it reports 0 via the new bitmap_if_materialized.

Propagation

Materializing on demand can fail, so Result propagates from RoaringFlagsRead::{get_bitmap, get, iter_trues, iter_falses, count_trues, count_falses} through PayloadFieldIndexRead::count_indexed_points, FieldIndexRead::{get_telemetry_data, values_count, values_is_empty, value_retriever}, PayloadIndexRead::{indexed_points, get_telemetry_data}, build_info / build_telemetry, and SegmentEntry::{info, get_telemetry_data} — out into shard, edge (including the public EdgeShardRead::info) and collection. Most consumers were already in OperationResult contexts.

bool_index/read_ops.rs::value_retriever must return an infallible per-point closure, so both bitmaps are resolved once at construction rather than per point.

Two collection sites read info() for size fields only; they now call the infallible size_info(). That is strictly better: /telemetry no longer forces every bool bitmap into memory.

Behavior change worth reviewing

ProxySegment::size_info() used to delegate to info(), so it uniquely returned a populated index_schema. To keep it infallible the proxy adjustments moved into adjusted_info(), fed by the wrapped segment's size_info(), which by contract leaves index_schema empty (read_view/info.rs:27). The proxy now matches plain Segment::size_info(). No caller reads index_schema off size_info(), but this is a semantic change, not a pure refactor.

Test plan

The existing live_reload_matches_fresh_open tests never read the index before reloading, so once open stopped materializing, they stopped covering the in-place delta path entirely — I disabled live_reload's deletes and appends and both still passed. Each is now split into _materialized / _lazy variants; the sabotage fails the _materialized pair.

Each half of the new live_reload contract is pinned independently, verified by breaking it:

• making refresh_counts a no-op (stale counts survive) fails live_reload_matches_fresh_open_materialized
• making it always recompute (forcing a scan on an untouched index) fails live_reload_matches_fresh_open_lazy

New open_does_not_materialize_bitmaps pins the laziness itself — nothing else in the module would notice an eager open, since every other test reads the index.

• cargo test -p segment --lib — 867 passed, 0 failed
• cargo test -p collection --lib — 244 passed, 0 failed
• cargo test -p shard --lib / -p edge --lib — 35 / 86 passed
• cargo clippy --workspace --all-targets — clean

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 900932ac-48a4-4619-875c-8555af9f01d7

📥 Commits

Reviewing files that changed from the base of the PR and between d7c4307 and f386795.

📒 Files selected for processing (5)

• lib/edge/publish/examples/src/bin/add-named-vector.rs
• lib/edge/publish/examples/src/bin/bm25-search.rs
• lib/edge/publish/examples/src/bin/demo.rs
• lib/edge/publish/examples/src/bin/facet_test.rs
• lib/edge/publish/examples/src/bin/restore-snapshot.rs

✅ Files skipped from review due to trivial changes (1)

• lib/edge/publish/examples/src/bin/demo.rs

---

📝 Walkthrough

Walkthrough

This PR changes segment, shard, and field-index read APIs to return OperationResult in many places, including segment info, telemetry, indexed-point counts, value retrievers, and payload-index read helpers. It also adds lazy bitmap/count materialization with OnceLock for read-only roaring flags and read-only bool indexes, updates live-reload behavior to refresh only materialized state, and adjusts collection, shard, edge, proxy, and test call sites to unwrap or propagate the new fallible results.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• qdrant/qdrant#8966: Changes the same PayloadFieldIndexRead count/filter/telemetry read surface that is expanded here to use OperationResult.
• qdrant/qdrant#9236: Touches ReadOnlyRoaringFlags in the same module, which this PR extends with lazy bitmap materialization.
• qdrant/qdrant#9398: Updates the same segment read-entry APIs so info() and telemetry return OperationResult.

Suggested reviewers: coszio, dancixx

Poem

I hopped through bitmaps, soft and slow,
Until they woke when counts would show.
No more surprise scans in the night,
Just Results gleaming, crisp and bright. 🐰

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: lazy roaring flags bitmaps and bool index counts.
Description check	✅ Passed	The description accurately summarizes the lazy bitmap/count changes, propagation, proxy semantic change, and test updates.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coszio]

Looks good, it was a crazy amount of changes to consider the OnceLocks.

It would be nice to propagate the Populate option through preopen and open. We can do it outside of this PR.

[generall]

It would be nice to propagate the Populate option through preopen and open. We can do it outside of this PR.

yes, I think it should be part of query-specific load profile