Skip to content

feat(vector/search): Stage-2 rerank for Flat/IVF via the shared pipeline (#650 PR-2) - #936

Merged
mosuka merged 1 commit into
mainfrom
feat/650-flat-ivf-rerank
Aug 2, 2026
Merged

feat(vector/search): Stage-2 rerank for Flat/IVF via the shared pipeline (#650 PR-2)#936
mosuka merged 1 commit into
mainfrom
feat/650-flat-ivf-rerank

Conversation

@mosuka

@mosuka mosuka commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

PR-2 of the #650 rerank-pipeline campaign: Stage-2 rerank (Issue #481) now works end-to-end on Flat and IVF through the PR-1 RerankPipeline — previously both returned NotImplemented for rerank_factor, with all three layers (emit / load / consume) missing.

Changes

  • Config wiring gap (found by the e2e test): FlatIndexConfig/IvfIndexConfig always had a rerank_storage field, but VectorStore::extract_index_type_config never mapped it from the schema option (only HNSW's from_hnsw_option did) — so nothing downstream could ever fire. Both arms now map it (the bug(vector/store): HnswOption::rerank_storage is never propagated to HnswIndexConfig — Stage-2 sidecar is not emitted via VectorStore/Engine #790 defect class: an option dropped at the config seam).
  • Writers: mirror HNSW's sidecar block — temp+rename atomicity, payload in each writer's exact record-emission order (Flat: pre-sorted self.vectors; IVF: the cluster-grouped flatten order via the already-materialized all_vectors) so the reader's position mapping is the identity. Also removed a pre-existing duplicated comment in the IVF writer.
  • Readers: new shared rerank_sidecar::load_rerank_sidecar helper (generalized from the HNSW reader's inline block): Eager-only, lenient-if-absent, loud on dim/vector_count mismatch. Both readers gain the pool field + accessor.
  • Searchers: the upfront NotImplemented is gone. Field-routed queries with a loaded sidecar run the sorted quantized candidates through a one-stage RerankPipeline (F32SidecarStage); survivors carry exact f32 distances and stamp score_basis = "f32-rerank" (bug(vector/search): cross-segment scores are not comparable — out-of-range queries clamp to similarity 1.0 per segment #927 fan-out contract — segmented layouts behave correctly for free). Otherwise rerank_factor silently falls back to Stage 1, the same convention as HNSW's missing-sidecar path. include_vectors refetches vectors only for the final top-k on the rerank path.
  • Docs (EN+JA, 6 files): the "Flat / IVF … do not yet write or consume the sidecar" caveats replaced with three-index-type support notes (schema_format.md, grpc_api.md, vector_indexing.md).

Tests

  • New vector_rerank_flat_ivf_test.rs (4 e2e mirrors of the HNSW suite): sidecar emitted at segment_00000N.{flat,ivf}.f32, doc1 wins, rerank score differs bit-wise from the int8 score (proving the f32 pool was used); the silent Stage-1 fallback on sidecar-less fields now succeeds (pinning the NotImplemented removal); a multi-commit segmented Flat case exercises the fan-out's exact-basis skip.
  • RED-proved: disabling the searcher-side consumption fails the "score changed by rerank" halves; restore is green.
  • The IVF record-order/sidecar-order identity is verified empirically by the exact-score assertions (a mapping skew would rescore against the wrong vectors and fail them).

Verification

  • cargo fmt --check and cargo clippy --all-targets -- -D warnings clean on stable and 1.97.0
  • cargo test -p laurus --lib: 1261 passed; --tests: all 67 binaries ok
  • wasm32 cargo check -p laurus-wasm: clean; cargo doc --no-deps -p laurus: 73 warnings (baseline)
  • markdownlint-cli2: 0 errors; both mdBooks build

Closes #932. Part of #650; depends on #931 (merged as #935). Leaves #673 (PR-3: SQ sidecar + 3-stage chain) as the campaign's remaining child.

@mosuka
mosuka merged commit 6668b71 into main Aug 2, 2026
22 checks passed
@mosuka
mosuka deleted the feat/650-flat-ivf-rerank branch August 2, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(vector/search): Stage-2 rerank for Flat/IVF via the shared pipeline (#650 PR-2)

1 participant