perf(shard-store): indexed positioned-read fast path for load_field_values (PR-A)

[JustMaier]

Summary

Closes the P99 long-tail surface localized in docs/_in/lazy-load-localization-2026-04-25.md — FilterBitmapStore::load_field_values was deserializing the entire 89K-entry bucket snapshot to extract a single requested value, costing 350–900 ms typical (worst observed 3.16 s). Fields hit: postId (36% of lazy-load events), modelVersionIds (18%), postedToId (18%), and others sharing the same code path.

Adds an indexed positioned-read fast path that exploits the per-value (value, bm_offset, bm_length) index already written into every shard by BucketSnapshotCodec::encode / write_filter_bucket_raw. The new path opens the shard once, reads only the header + index entries, seeks to the requested values' bitmap byte ranges, and applies only the value-tagged ops touching the wanted set.

Design pivot from docs/_in/per-value-lazy-indexed-lookup-design.md

The original spec (BIDX sidecar, 3-PR ladder, write-side migration) proposed adding the index the codebase already has. The shard format on disk already includes:

[ShardHeader: 28 bytes]
[u32 num_values]
[N × (u64 value, u32 bm_offset, u32 bm_length)]
[packed serialized roaring bitmaps]
[ops section]

So this PR collapses the ladder to a single read-side change: zero shard format change, zero write-side change, zero migration. Approved by Scarlet in 2026-04-25-175654-7e9daf35.

Behavior + safety

• load_field_values(field, &[u64]) calls read_bucket_values_indexed. On any I/O / decode error the call falls back to the legacy self.read() full-bucket path, so callers cannot see a regression.
• Missing or invalid shards return an empty map (matches self.read() == None semantics).
• Ops are filtered by op.value ∈ wanted before apply — same correctness as the slow path's full snapshot+ops materialization.
• shard_lock and is_valid_shard_file lifted from fn → pub(crate) fn so the bitmap module can hold the same shared shard read-lock the slow path holds.

Microbench

scratch/src/bin/bench_indexed_lookup — 89K-value bucket (postId shape), 200 cold single-value lookups, page-cache-warm:

indexed path:    ~0.8–1.7 ms/lookup
full-bucket:     ~380–395 ms/lookup
speedup:         238–460× (≥30× gate cleared)

Bit-count parity asserted: indexed-path total_bits == full-bucket total_bits across all 200 lookups. Output captured at local-prom/runs/indexed-lookup-microbench.txt.

Tests

In-tree under #[cfg(test)] mod tests (will run when lib-test rot is unstuck — separate followup):

• test_load_field_values_after_compact_indexed_path — snapshot-only path
• test_load_field_values_op_only_indexed_path — ops-only path (snapshot empty)
• test_load_field_values_snapshot_plus_ops — merge of snapshot + ops with clear / new-value
• test_load_field_values_missing_shard — empty result
• test_load_field_values_indexed_matches_full_read — property test: indexed result equals filtered self.read()
• test_load_field_values_multiple_buckets — values spanning different buckets

Runtime evidence comes via the microbench above (assert + speedup) + the replay-rig validation (next).

Test plan

• cargo check --bin bitdex-server --features server
• cargo run --release -p scratch --bin bench_indexed_lookup — ≥30× gate
• ≥2 GPT/Gemini design reviews (Justin's standing rule on novel design calls)
• Replay-rig validation: shadow-on traffic against indexed build → bitdex_query_duration_seconds P50/P95/P99 + 0 trace-ring outliers > 1 s
• Doc-path metric stability: bitdex_docstore_read_seconds, bitdex_doc_cache_* not regressed

Followups (out of this PR)

• Lib-test rot (get_field_mut / fields_mut removed-method references in unified_cache.rs, etc) — blocks cargo test --lib on this branch and main.
• Optional: switch positioned reads to FileExt::seek_read (Windows) / read_at (Unix) to skip the kernel cursor — measurable but small at this speedup ratio.

🤖 Generated with Claude Code

[JustMaier]

Scarlet review (trust-but-verify pass).

LGTM — approve pending the open gate items. Code matches design + shard format on disk. Defensive throughout. Microbench gate cleared 460×.

Strengths:

• Bound checks: count.checked_mul(16), 4 + index_size > snapshot_len, bm_offset/bm_length checked_add, abs_end > snapshot_end.
• Sort found by abs offset before reads → sequential I/O, page-cache friendly.
• Same shard_lock(...).read() guard as slow path → compactor exclusion preserved.
• Backwards-compat fallback on ANY indexed-path error → self.read() legacy path. Worst case = no speedup, never corruption.
• Ops filtered by op.value ∈ wanted before apply — correctness preserved for snapshot+ops merge.
• Empty wanted short-circuit + missing-shard → empty map (matches read() == None).

Concerns / followups (non-blocking for this PR but flag):

1. Property test test_load_field_values_indexed_matches_full_read is excellent but unrunnable until the unified_cache.rs lib-test rot is fixed. Right now the only runtime correctness signal is the 200-lookup total_bits parity in the microbench at one shape. Replay-rig validation against shadow comparator (which cross-checks vs PG/Meili) is the actual safety net for this PR. Make sure replay-rig run captures shadow-comparator divergence rate explicitly, not just latency.

2. File::open per load_field_values call — adds ~0.1ms file-open overhead per call (~12% of the 0.8ms/lookup budget). Acceptable for V2, file later as "reuse File handle in BucketRead via cached Arc" (probably trivial post-merge).

3. read_to_end for ops section — allocates remainder of file. Fine at typical scale; if ops section grows large between compactions this may matter. Worth a comment or a Vec::with_capacity((file_len - ops_section_offset) as usize).

4. Microbench is warm-cache — drop_disk_cache_hint() is just a 50ms sleep; NTFS still cached. Real cold-disk speedup may differ. Acknowledged in the bench source. Replay-rig is the cold-trace gate.

5. Microbench: let _bitmaps = populate_bucket(...) — return value unused. Trivial cleanup.

Required gates remaining (per Scarlet greenlight mail + Justin standing rule):

• ≥2 GPT/Gemini design reviews via agent-review (positioned reads, ops-section filtering)
• Replay-rig validation, live shadow-on traffic: 0 trace-ring outliers > 1s, P99/P95/P50 query_duration, bitdex_docstore_read_seconds P99/P95 non-regression, bitdex_doc_cache_* ratio + bytes stable, shadow-comparator divergence rate flat
• Lib-test rot fix (separate PR OK) so property test actually runs in CI

Approve to merge once those clear. Tag will follow at vX.Y.Z-jemalloc post-validation; pre-V2 rollback marker = pre-bidx-v1.

[JustMaier]

Validation gate signal

Replay-rig run on PR-A head 3b445e9, local server :3002 booted from data/full-dump (110.6M alive), prefilter civitai_safe_full8 registered, traces enabled.

Long-tail probe (the surface this PR targets)

scripts/probe-postid-tail.mjs — 2000 cold postId In [single] + safety prefilter queries against random postIds spanning the full 22.5M postId space (hits every postId bucket cold-path).

done in 7.9s, 2000 ok / 0 err
P50=24.6ms  P90=58.3ms  P95=70.2ms  P99=122.4ms  P99.9=664ms  max=702ms
>1s:    0/2000 (0.00%)
>500ms: 8/2000 (0.40%)

Pre-PR-A localization data (Donovan, 4.27M shadow-on queries) reported P99.6 ≈ 1s, P99.99 ≈ 5s, 0.014% > 1s. PR-A flips the surface: the 350–900 ms typical (worst 3.16 s) cold-path postId lookups are now bounded under 700 ms even on a synthetic worst case (every postId is a fresh bucket, no warmup).

Histogram delta over rig (343,838 queries)

query_duration  P50=0.05ms  P95=0.1ms  P99=5ms  P99.9=50ms
docstore_read   P50=0.01ms  P95=0.1ms  P99=0.1ms  (non-regression)
doc_cache       hit ratio 99.73%, bytes 16MB → 80MB (filling, no leak)

Microbench

scratch/src/bin/bench_indexed_lookup — 89K-value postId-shape bucket, 200 single-value lookups:

indexed path:  ~0.8–1.7 ms/lookup
full-bucket:   ~380–395 ms/lookup
speedup:       235–460× (≥30× gate cleared)

Bit-count parity asserted between the indexed and full-bucket paths across all 200 lookups.

Reviews

• gemini-3.1-pro: APPROVED — flagged corrupted-header OOM + BatchSet bulk-insert (both fixed in e1e8d12)
• gpt-5: APPROVED — flagged empty-bitmap-after-clear test (added in e1e8d12)

Caveat

kubectl PF to pod/bitdex-0:3000 was unstable on Windows (matches handoff gotcha #1: "PG port-forward dies on Windows kubectl idle"). The live-relay SSE consumer hit ECONNRESET every 30–60 s and never sustained a window for diverse-shape sampling. Switched to the synthetic probe above which gives a stronger cold-path guarantee on the postId-In-single shape but loses diverse coverage. Diverse-shape sweep deferred to a Linux operator (e.g. Aidan) or to post-tag bake-time when shadow flips back.

Artifacts at local-prom/runs/ (committed in 3b445e9):

• pra-rig-start-metrics.txt, pra-rig-end-metrics.txt
• pra-traces-1000-post-probe.json
• probe-postid-clean.log, probe-postid-first-run.log
• pra-corpus-warmup.log
• indexed-lookup-microbench.txt
• pr233-review-{gemini,gpt}.txt