feat(vector/search): Stage-2 rerank for Flat/IVF via the shared pipeline (#650 PR-2) - #936
Merged
Conversation
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR-2 of the #650 rerank-pipeline campaign: Stage-2 rerank (Issue #481) now works end-to-end on Flat and IVF through the PR-1
RerankPipeline— previously both returnedNotImplementedforrerank_factor, with all three layers (emit / load / consume) missing.Changes
FlatIndexConfig/IvfIndexConfigalways had arerank_storagefield, butVectorStore::extract_index_type_confignever mapped it from the schema option (only HNSW'sfrom_hnsw_optiondid) — so nothing downstream could ever fire. Both arms now map it (the bug(vector/store): HnswOption::rerank_storage is never propagated to HnswIndexConfig — Stage-2 sidecar is not emitted via VectorStore/Engine #790 defect class: an option dropped at the config seam).self.vectors; IVF: the cluster-grouped flatten order via the already-materializedall_vectors) so the reader's position mapping is the identity. Also removed a pre-existing duplicated comment in the IVF writer.rerank_sidecar::load_rerank_sidecarhelper (generalized from the HNSW reader's inline block): Eager-only, lenient-if-absent, loud on dim/vector_count mismatch. Both readers gain the pool field + accessor.NotImplementedis gone. Field-routed queries with a loaded sidecar run the sorted quantized candidates through a one-stageRerankPipeline(F32SidecarStage); survivors carry exact f32 distances and stampscore_basis = "f32-rerank"(bug(vector/search): cross-segment scores are not comparable — out-of-range queries clamp to similarity 1.0 per segment #927 fan-out contract — segmented layouts behave correctly for free). Otherwisererank_factorsilently falls back to Stage 1, the same convention as HNSW's missing-sidecar path.include_vectorsrefetches vectors only for the final top-k on the rerank path.schema_format.md,grpc_api.md,vector_indexing.md).Tests
vector_rerank_flat_ivf_test.rs(4 e2e mirrors of the HNSW suite): sidecar emitted atsegment_00000N.{flat,ivf}.f32, doc1 wins, rerank score differs bit-wise from the int8 score (proving the f32 pool was used); the silent Stage-1 fallback on sidecar-less fields now succeeds (pinning theNotImplementedremoval); a multi-commit segmented Flat case exercises the fan-out's exact-basis skip.Verification
cargo fmt --checkandcargo clippy --all-targets -- -D warningsclean on stable and 1.97.0cargo test -p laurus --lib: 1261 passed;--tests: all 67 binaries okcargo check -p laurus-wasm: clean;cargo doc --no-deps -p laurus: 73 warnings (baseline)markdownlint-cli2: 0 errors; both mdBooks buildCloses #932. Part of #650; depends on #931 (merged as #935). Leaves #673 (PR-3: SQ sidecar + 3-stage chain) as the campaign's remaining child.