feat(embedding): version the vector space so an embedder swap is safe - #327
Merged
Conversation
SMK1705
commented
Jul 24, 2026
SMK1705
commented
Owner
…#316) Groundwork for the neural embedder. Today every stored vector is an anonymous BLOB: note_embeddings is (noteId, vector) with nothing recording WHICH embedder produced it. Drop a neural embedder in behind the Embedder seam and the index would silently cosine a 768-dim neural query against 256-dim lexical vectors — not a crash, just quietly meaningless similarity scores across search, Ask recall and near-duplicate merging. That has to be fixed BEFORE a model lands, not after. - Embedder gains `id` (its vector-space identity) and `searchFloor`. The floor was a single global constant tuned for the hashing space; it is calibrated per space, so it now travels with the embedder. All four call sites read it from the index instead. - note_embeddings gains `embedderId` (migration 18→19, defaulting to hashing-v1 — which is in fact what produced every existing row, so an upgrade re-indexes nothing). - SemanticIndex.index() tags what it writes; scores() only reads vectors from the ACTIVE embedder (plus a dimension guard), so a foreign-space vector is skipped rather than scored; backfill() treats a vector from another embedder as absent, so the backfill the app ALREADY runs on Ask/Notes launch re-indexes after a swap with no extra step. Net effect: swapping the embedder becomes a one-line DI change that heals itself, and a half-finished re-index degrades recall instead of inventing nonsense similarities. Tests: SemanticIndexTest covers tagging, foreign-space vectors being skipped, re-index after a swap (asserting the new dimension lands), and the per-embedder floor, using a stand-in embedder with a different id/dimension/floor. MigrationTest registers 18→19 in the full chain (so Room's schema validation covers it) and adds a targeted test that an existing vector survives with its bytes intact and is tagged hashing-v1. Does NOT add a model or a runtime — that's the follow-up, and it now has somewhere safe to land.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.