fix(vector/index): segmented search/merge dedup, sealed-segment deletes, PQ min-train policy (#634 PR-2)

[mosuka]

PR-2 of the #634 segment-per-commit campaign (inert — fixes correctness holes in the unwired segmented machinery).

Scope

• Newest-segment-wins dedup in multi-segment search: cross-segment duplicates of the same doc_id currently double-count (segmented_field.rs:500-521 does entry.score += hit.score)
• Wire merge dedup: merge_vectors (newest-wins) is dead code (merge_engine.rs:245-266) — connect it to merge_segments
• Sealed-segment deletes: SegmentedVectorField::delete_document silently no-ops for docs in sealed segments — route through the global DeletionBitmap + persist_deletions + set has_deletions on covering segments
• PQ policy: k-means on a tiny segment is meaningless — min-train guard (small segments stay Scalar8Bit/f32), PQ trained at merge time from the lossless f32 sidecar (enhancement(vector/hnsw): merged-segment rerank sidecar is rebuilt from dequantized int8, not the source sidecars' original f32 #795)

Acceptance criteria

• Red/green test per fix (disabling each fix fails its test)
• WAL-replay-shaped same-doc_id upsert across segments returns the newest version exactly once
• Sealed-segment delete is search-invisible after flush and survives merge

Refs #634

[mosuka]

Implementation report (PR-2)

Implements the four scoped fixes plus 14 adversarial-review findings (two review rounds, 24 agents total; every confirmed finding fixed and negative-tested).

Scoped fixes

1. Search newest-source-wins dedup — cross-segment same-doc_id copies were scored once per copy (score +=), letting stale data outrank live data.
2. Merge newest-wins dedup — merge_segments collapses (doc_id, field) duplicates newest-generation-first (duplicates_removed stat; dead merge_vectors helper removed).
3. Sealed-segment deletes — previously a silent no-op; now marks the shared DeletionBitmap (+ has_deletions flags), with new DeletionBitmap::undelete_document completing the same-id upsert dance.
4. PQ min-train guard — PQ-configured segments below the k-means centroid count are written as Scalar8Bit (self-describing LVS1 header) instead of training a degenerate, oversized codebook.

Review round 1 (6 confirmed → fixed)

• CRITICAL — merged segments registered with generation 0 (live-repro'd: the next merge permanently dropped the newest copy as a "duplicate"): now registers the engine's info inheriting max(source generations) (not max+1 — that would launder stale copies above untouched newer segments), plus a generation-gap expansion so merge candidates are generation-contiguous, and apply_merge measures real on-storage size.
• HIGH — returned-hits masking leaks stale copies (live-repro'd: after an upsert, old-embedding queries rarely rank the new copy into its segment's top-k): masking is now containment-based — a sealed hit is dropped when the active buffer (HnswIndexWriter::contains_doc, new) or any newer-generation reader (VectorStorage::contains) holds the doc; one read guard spans the request.
• MEDIUM — bitmap flips not atomic with buffer mutations (lost-delete race): both run under the active_segment write lock (bitmap first on delete).
• Legacy all-zero-generation manifests stamped in list order at load; the cross-field shared-bitmap hazard documented as a per-field-bitmap requirement for PR-3.

Review round 2 (8 confirmed → fixed)

• HIGH — the pq-fastscan feature test suite actually failed (neither the PR verification nor CI enables that feature — caught by a reviewer running it): FastScan gets its own FASTSCAN_MIN_TRAIN_VECTORS = 16 (it trains K=16 centroids), and the partial-block test moves n=5 → 21 (padding coverage kept).
• MEDIUM — FastScan integration coverage restored by the per-variant threshold; mixed manifests (a PR-1-era zero-generation merged segment among stamped ones) stamp the zero entry as oldest (the loss-minimizing direction); SimpleMergePolicy now selects a generation-contiguous window (the old global-smallest selection amplified merges through the gap expansion).
• LOW — small-corpus PQ tests un-vacated (core test + all four binding tests grown to 256 vectors); sealed-segment search over-fetches 2× so masked stale copies don't consume result slots; sealed readers load before the active guard (no I/O under the lock); the PQ fallback documented in vector_indexing.md (EN/JA).
• Self-found while fixing: the guard skipped PQ geometry validation (subvector_count divides dimension) for small segments — validation now runs in the guard arms, so invalid configs are rejected at commit regardless of corpus size (the binding reject-tests' semantics).

Verification

• 1243 lib + 1428 integration + 1562 --features pq-fastscan tests green (the feature suite is now part of the verification set — recorded as a process lesson).
• fmt + clippy on 1.95/1.97, fastscan clippy, wasm clippy: clean; markdownlint: 0.
• 8 negative verifications: disabling each load-bearing fix makes its regression test fail.