diff --git a/CLAUDE.md b/CLAUDE.md index 0eac446..0c5984f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,7 +21,7 @@ schema must satisfy the data model, never the reverse. - `planning/vision-and-scope.md` — the *why*: principles, the two design tenets, v1 scope, locked decisions. - `planning/data-model.md` — the *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary. -- `planning/index-engine.md` + `planning/specs/completed/index-engine-build.md` — the *how*: SQLite (FTS5 + `sqlite-vec`) projection, table DDL, the build order, data flows. +- `planning/index-engine.md` + `planning/specs/completed/index-engine-build.md` — the *how*: SQLite (FTS5 + in-process vector scan; `sqlite-vec` until 2026-07-12, see `research/discovery-scan-strategy.md`) projection, table DDL, the build order, data flows. - `planning/tasks.md` — the working queue (what's done, what's next). **Read this first to know current state.** Planned-but-unstarted work lives in GitHub Issues; shipped build specs live in `planning/specs/completed/` (index engine, desktop MVP, async indexing, projection/embedding split, desktop editing, live preview). @@ -95,7 +95,7 @@ so Tauri/wry tracing doesn't pollute the file (an explicit `B2_LOG` is honored v 1. **Markdown files** (`/*.md`) — the source of truth, plain and portable. Every committed connection lives here: a body `[[link]]`, or a frontmatter `relations:` entry (written by `b2 link`). -2. **Disposable SQLite index** (`/.b2/b2.sqlite`) — FTS5 + `sqlite-vec` + the typed `edges` graph. +2. **Disposable SQLite index** (`/.b2/b2.sqlite`) — FTS5 + plain-table vectors (`embeddings` + `note_centroids`, scored in-process) + the typed `edges` graph. Drop it and `reindex` rebuilds it identical. Nothing here is authoritative, and **no durable state lives outside the Markdown.** @@ -127,7 +127,8 @@ no model) + `b2 link` (the human commits). A reranker would be the next seam if/ - **`b2-core`** — the whole index engine and the typed `Vault` façade. Deliberately **model-free** (no candle) so its test suite stays fast and deterministic. Deps: rusqlite (bundled SQLite + FTS5), - `sqlite-vec`, blake3, ulid, yaml-rust2. + blake3, ulid, yaml-rust2. (No vector extension: vectors are plain BLOB tables scored in-process — + `embed::l2_sq` over `db::for_each_stored_vector`/`for_each_note_centroid` — since #38.) - **`b2-embed`** — the real candle-backed embedder. Heavy ML deps (candle, tokenizers, hf-hub) live **only here**. `provision` (`b2 init`) downloads + verifies the model into a shared XDG cache; `LocalEmbedder::load` fails fast with "run `b2 init`" if absent. @@ -159,12 +160,15 @@ adapters wire the real model. (`specs/completed/projection-embedding-split.md`): model-free `project_vault` (notes/chunks/FTS/edges) and `embed_vault` (fills the DB-derived missing-vector set); `reindex` composes them, and `search` falls back to BM25-only on a projected-but-unembedded vault. -- **Flow ② hybrid search** (`search.rs`) — BM25 (`chunks_fts`) ⊕ vector KNN (`chunks_vec`) fused with - Reciprocal Rank Fusion (k=60), resolved from chunks up to notes. Raw NL queries are sanitized into a - safe FTS5 `MATCH` expression (punctuation is FTS5 syntax and would otherwise crash the parse). +- **Flow ② hybrid search** (`search.rs`) — BM25 (`chunks_fts`) ⊕ vector KNN (an exact in-process scan + of `embeddings`) fused with Reciprocal Rank Fusion (k=60), resolved from chunks up to notes. Raw NL + queries are sanitized into a safe FTS5 `MATCH` expression (punctuation is FTS5 syntax and would + otherwise crash the parse). - **Flow ③ connection discovery** — **`b2 similar`** (`discover::candidates`) surfaces the semantically - nearest *unlinked* notes (vector KNN over stored embeddings, minus the anchor's 1-hop graph neighbors — - no model call); **`b2 link`** appends a typed `relations:` entry to the source note's frontmatter + nearest *unlinked* notes in **two stages** (#38): a coarse O(notes) scan over per-note centroids + (`note_centroids`, maintained by the embed pass) shortlists candidates, then exact max-sim over only + the shortlist's chunk vectors — minus the anchor's 1-hop graph neighbors, no model call; + **`b2 link`** appends a typed `relations:` entry to the source note's frontmatter (`note::add_relation`, Markdown-first, **never the body**) and re-projects it as an `origin=frontmatter` active edge. No suggestion queue — a connection exists only once you author it. - **`graph_filtered_search`** (`search.rs`) — the vector⨝graph join: nearest chunks whose note is @@ -183,12 +187,17 @@ display-only. ### Embedding-space discipline -`chunks_vec` (a `vec0` virtual table) is created at **embed time**, not in the base migration, because -its dimension is a DDL literal pinned to the embedder's `dim`. `meta` records `(embed_model_id, -embed_dim)` — the only place a model swap is detectable. A swap drops `chunks_vec` and re-embeds on +Vectors live in **plain tables** — `embeddings(chunk_id, vector)` and `note_centroids(note_b2id, +centroid)` — created at **embed time**, not in the base migration: their existence is the "this vault +has an embedding space" signal the projected-but-unembedded fallbacks key on. Every distance is +computed **in-process** (`embed::l2_sq`, one sequential scan statement — the former `sqlite-vec` +`vec0` table charged a per-row shadow probe on every scan, #36/#38). `meta` records `(embed_model_id, +embed_dim)` — the only place a model swap is detectable. A swap drops both tables and re-embeds on `reindex`; `search` **fails fast** on a mismatch rather than returning silently-wrong results. `open` never mutates the vector space (so changing the configured model can't wipe vectors on the next -command). +command). Centroids are derived data with the vectors' own lifecycle: the embed pass refreshes a +note's centroid after filling its vectors; a re-chunk drops it (`db::replace_chunks`) — no separate +invalidation exists or is needed. ## Conventions @@ -222,6 +231,6 @@ command). - Signatures: accept `&str` not `&String`, `&[T]` not `&Vec`. Return owned types and let callers borrow. - Prefer iterator chains over manual index loops (`for x in &items`, not `for i in 0..items.len()`). - Do NOT introduce `async`/`tokio`, generics, traits, or macros until there's a concrete need. No speculative abstraction. -- `unsafe` requires an explicit `// SAFETY:` comment stating the invariant that makes it sound (see `db.rs`'s `sqlite-vec` registration and `model.rs`'s weights mmap); otherwise disallowed. +- `unsafe` requires an explicit `// SAFETY:` comment stating the invariant that makes it sound (see `model.rs`'s weights mmap); otherwise disallowed. - Derive `Debug` on public data types (and `Clone`/`PartialEq` where it makes sense). - Keep modules small and domain-named; document public items with `///` comments stating intent, not mechanics. diff --git a/Cargo.lock b/Cargo.lock index db477ef..0ed4c1a 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -197,7 +197,6 @@ dependencies = [ "rusqlite", "serde", "serde_json", - "sqlite-vec", "tempfile", "thiserror 2.0.18", "tracing", @@ -4342,15 +4341,6 @@ dependencies = [ "unicode-segmentation", ] -[[package]] -name = "sqlite-vec" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0ba424237a9a5db2f6071f193319e2b6a32f7f3961debb2fbbfe67067abce3f" -dependencies = [ - "cc", -] - [[package]] name = "sqlite-wasm-rs" version = "0.5.5" diff --git a/crates/b2-core/Cargo.toml b/crates/b2-core/Cargo.toml index ab098e2..942b102 100644 --- a/crates/b2-core/Cargo.toml +++ b/crates/b2-core/Cargo.toml @@ -12,7 +12,6 @@ blake3 = "1.8.5" rusqlite = { version = "0.40.1", features = ["bundled", "trace"] } serde = { version = "1.0.228", features = ["derive"] } serde_json = "1.0.150" -sqlite-vec = "0.1.9" thiserror = "2.0.18" # Structured debug logging (facade only — emitting events costs nothing until an # adapter installs a subscriber; the CLI's is opt-in via B2_LOG/B2_DEBUG). diff --git a/crates/b2-core/src/db.rs b/crates/b2-core/src/db.rs index a6506cd..5a85773 100644 --- a/crates/b2-core/src/db.rs +++ b/crates/b2-core/src/db.rs @@ -1,22 +1,27 @@ //! Opening the index, the schema migration, and the projection helpers for the -//! Markdown-derived tiers: `notes`/`note_aliases`, `chunks` (+FTS5), and the -//! typed `edges` graph, plus the `b2id ⇄ path` resolver. +//! Markdown-derived tiers: `notes`/`note_aliases`, `chunks` (+FTS5), the +//! `embeddings`/`note_centroids` vector tables, and the typed `edges` graph, plus +//! the `b2id ⇄ path` resolver. //! -//! `sqlite-vec` is registered as a SQLite *auto-extension* (statically linked, no -//! runtime `load_extension`), and every connection is opened `WAL` + -//! `foreign_keys=ON` per planning/specs/completed/index-engine-build.md §0. Every table here -//! is a derived projection of `Markdown` — nothing is a source of truth. +//! Every connection is opened `WAL` + `foreign_keys=ON` per +//! planning/specs/completed/index-engine-build.md §0. Every table here is a derived +//! projection of `Markdown` — nothing is a source of truth. +//! +//! Vectors live in **plain tables** and every distance is computed in-process +//! (schema v3, #38). The previous store — `sqlite-vec`'s `chunks_vec` `vec0` +//! virtual table — charged a per-row shadow-table probe on every scan (~38.6k +//! internal statements per `b2 similar` on a real vault) while its only shipped +//! search was the same brute force we compute ourselves; a plain-table scan is one +//! sequential statement. use crate::chunk::Chunk; use crate::embed::pack_f32; use crate::error::Result; use rusqlite::trace::{TraceEvent, TraceEventCodes}; -use rusqlite::{ffi, params, Connection, OptionalExtension, StatementStatus}; -use sqlite_vec::sqlite3_vec_init; +use rusqlite::{params, Connection, OptionalExtension, StatementStatus}; use std::collections::HashMap; -use std::os::raw::{c_char, c_int}; use std::path::Path; -use std::sync::{Once, OnceLock}; +use std::sync::OnceLock; use std::time::Duration; /// The B2 index schema version stamped into `meta.schema_version`. Bumping it is @@ -24,28 +29,12 @@ use std::time::Duration; /// the next `reindex` rebuild them (the index is disposable). **2** dropped the /// suggestion machinery — the `status` column, the `origin='suggested'` value, and /// the `edge_provenance` table — with the 2026-07-04 relator cut (data-model.md §4). -pub const SCHEMA_VERSION: i64 = 2; - -static REGISTER_VEC: Once = Once::new(); - -/// Register `sqlite-vec` exactly once per process so every later `Connection` -/// exposes the `vec0` virtual table with no runtime `load_extension`. -fn register_sqlite_vec() { - // sqlite-vec and rusqlite each declare their own (ABI-identical) SQLite FFI - // types, so the init fn must be transmuted to the signature rusqlite's - // `sqlite3_auto_extension` expects — this mirrors the official sqlite-vec Rust - // example. The explicit annotation is the type clippy would otherwise ask for. - type InitFn = unsafe extern "C" fn( - *mut ffi::sqlite3, - *mut *mut c_char, - *const ffi::sqlite3_api_routines, - ) -> c_int; - REGISTER_VEC.call_once(|| unsafe { - ffi::sqlite3_auto_extension(Some(std::mem::transmute::<*const (), InitFn>( - sqlite3_vec_init as *const (), - ))); - }); -} +/// **3** replaced the `chunks_vec` vec0 virtual table with the plain `embeddings` + +/// `note_centroids` tables and dropped the `sqlite-vec` dependency (#38); a pre-3 +/// index's orphaned `chunks_vec` entry is left inert in `sqlite_master` (its module +/// is no longer linked, so it can't be dropped) — delete `.b2/b2.sqlite` for a +/// byte-clean slate; either way the next `reindex` rebuilds everything queried. +pub const SCHEMA_VERSION: i64 = 3; /// Statements at or over this take the slow-query WARN path (`B2_SLOW_QUERY_MS` /// overrides; see [`slow_query_threshold`]). @@ -120,7 +109,6 @@ fn on_sqlite_profile(event: TraceEvent<'_>) { /// Open (creating if needed) the B2 index at `path` with the locked pragmas and an /// idempotent migration. Safe to call on a fresh or an already-built index. pub fn open(path: &Path) -> Result { - register_sqlite_vec(); let conn = Connection::open(path)?; // Profile every statement on this connection through SQLite's trace_v2 hook — // the source of the `b2::sqlite` query-timing events (see `on_sqlite_profile`). @@ -128,15 +116,22 @@ pub fn open(path: &Path) -> Result { TraceEventCodes::SQLITE_TRACE_PROFILE, Some(on_sqlite_profile), ); - // execute_batch tolerates the row PRAGMA journal_mode returns. + // execute_batch tolerates the rows PRAGMA journal_mode / mmap_size return. // busy_timeout: WAL allows one writer at a time, and two short-statement // writers can now legitimately race (a save during the background embed — // desktop-editing.md §4). A modest wait turns that contention into a few-ms // stall instead of an immediate SQLITE_BUSY error. + // mmap_size + cache_size: the whole-space vector scans stream ~100+ MB of blob + // rows per call on a real vault; under the 2 MB default cache with no mmap that + // read path was pread/syscall-bound — the bulk of `b2 similar`'s ~4.4 s (#38). + // mmap_size is a *cap*, not an allocation (the OS page cache does the work — the + // one cache B2 is happy to lean on); cache_size is in KiB when negative (32 MiB). conn.execute_batch( "PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON; - PRAGMA busy_timeout = 5000;", + PRAGMA busy_timeout = 5000; + PRAGMA mmap_size = 1073741824; + PRAGMA cache_size = -32768;", )?; migrate(&conn)?; Ok(conn) @@ -145,7 +140,8 @@ pub fn open(path: &Path) -> Result { /// Create the schema and stamp `schema_version`. `IF NOT EXISTS` keeps the CREATEs a /// no-op on reopen; a `schema_version` mismatch drops the derived tables first so the /// next `reindex` rebuilds them (the index is disposable). The DDL mirrors -/// planning/specs/completed/index-engine-build.md §1 (chunks_vec is created at embed time). +/// planning/specs/completed/index-engine-build.md §1 (the vector tables are created at +/// embed time — see [`ensure_embedding_space`]). fn migrate(conn: &Connection) -> Result<()> { // `meta` must exist before we can read the schema version the index was built at. conn.execute_batch( @@ -164,11 +160,17 @@ fn migrate(conn: &Connection) -> Result<()> { ) .optional()? .and_then(|s| s.parse().ok()); + // The legacy vec0 `chunks_vec` (schema ≤ 2) is deliberately absent from this + // list: its module is no longer linked, so SQLite cannot DROP it — any orphaned + // entry stays inert in `sqlite_master` and nothing ever queries it (delete the + // index file for a byte-clean slate). `DELETE FROM meta` clears the recorded + // embedder, so the next embed pass recreates the vector tables from nothing. if prior.is_some_and(|v| v != SCHEMA_VERSION) { conn.execute_batch( "DROP TABLE IF EXISTS edge_provenance; DROP TABLE IF EXISTS edges; - DROP TABLE IF EXISTS chunks_vec; + DROP TABLE IF EXISTS note_centroids; + DROP TABLE IF EXISTS embeddings; DROP TABLE IF EXISTS chunks_fts; DROP TABLE IF EXISTS chunks; DROP TABLE IF EXISTS note_aliases; @@ -327,20 +329,20 @@ pub fn upsert_note(conn: &Connection, row: &NoteRow) -> Result<()> { // --------------------------------------------------------------------------- /// Replace a note's chunks (delete + reinsert) and return the new chunk ids in -/// `seq` order. The FTS triggers emit the `'delete'` sentinel for the removed -/// rows; `chunks_vec` has no FK/trigger back to `chunks`, so its stale rows are -/// cleared here explicitly. Together this is what makes an incremental re-index -/// equal a full rebuild. The caller embeds the returned ids (Flow ①). +/// `seq` order. The FTS triggers emit the `'delete'` sentinel for the removed rows, +/// and any stored vectors cascade with them (`embeddings.chunk_id` is an +/// `ON DELETE CASCADE` FK). The note's centroid summarizes the *old* chunk set, so +/// it is dropped here too — the next embed pass recomputes it. Together this is +/// what makes an incremental re-index equal a full rebuild. The caller embeds the +/// returned ids (Flow ①). pub fn replace_chunks(conn: &Connection, note_b2id: &str, chunks: &[Chunk]) -> Result> { + // Guarded on existence so the model-free projection pass still never *creates* + // the embedding space (projection-embedding-split.md §4). if embedding_space_exists(conn)? { - let old_ids: Vec = { - let mut stmt = conn.prepare("SELECT id FROM chunks WHERE note_b2id = ?1")?; - let rows = stmt.query_map([note_b2id], |r| r.get(0))?; - rows.collect::>>()? - }; - for id in old_ids { - conn.execute("DELETE FROM chunks_vec WHERE chunk_id = ?1", [id])?; - } + conn.execute( + "DELETE FROM note_centroids WHERE note_b2id = ?1", + [note_b2id], + )?; } conn.execute("DELETE FROM chunks WHERE note_b2id = ?1", [note_b2id])?; @@ -364,25 +366,28 @@ pub fn replace_chunks(conn: &Connection, note_b2id: &str, chunks: &[Chunk]) -> R } // --------------------------------------------------------------------------- -// embeddings — chunks_vec is created at the embedder's dim (not in migrate()), -// because the vec0 dimension is a DDL literal pinned to meta.embed_dim (§1.0). +// embeddings — the vector tables are created at embed time (not in migrate()): +// their *existence* is the "this vault has an embedding space" signal the +// projected-but-unembedded fallbacks key on (projection-embedding-split.md §5). // --------------------------------------------------------------------------- -/// Whether the `chunks_vec` virtual table currently exists. +/// Whether the embedding space (the `embeddings` table) currently exists. pub fn embedding_space_exists(conn: &Connection) -> Result { let n: i64 = conn.query_row( - "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'chunks_vec'", + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'embeddings'", [], |r| r.get(0), )?; Ok(n > 0) } -/// Ensure `chunks_vec` exists at `dim`, recording `(embed_model_id, embed_dim)` -/// in `meta`. If either differs from what is recorded — a model swap — the table -/// is dropped and recreated empty, so a full re-embed follows (index-engine.md -/// §8). `meta` is the only place a swap can be detected, so vectors never go -/// silently stale. +/// Ensure the vector tables (`embeddings` + `note_centroids`) exist, recording +/// `(embed_model_id, embed_dim)` in `meta`. If either differs from what is recorded +/// — a model swap — the tables are dropped and recreated empty, so a full re-embed +/// follows (index-engine.md §8). `meta` is the only place a swap can be detected, +/// so vectors never go silently stale. (`dim` is bookkeeping only now — a plain +/// BLOB column needs no `FLOAT[N]` DDL literal — but it still gates the swap and +/// the read-time fail-fast.) pub fn ensure_embedding_space(conn: &Connection, model_id: &str, dim: usize) -> Result<()> { let cur_model: Option = conn .query_row( @@ -403,21 +408,25 @@ pub fn ensure_embedding_space(conn: &Connection, model_id: &str, dim: usize) -> return Ok(()); } - // dim is an integer we control (never user input) → safe to interpolate. - conn.execute_batch(&format!( - "DROP TABLE IF EXISTS chunks_vec; - CREATE VIRTUAL TABLE chunks_vec USING vec0( - chunk_id INTEGER PRIMARY KEY, - embedding FLOAT[{dim}] - );" - ))?; + conn.execute_batch( + "DROP TABLE IF EXISTS note_centroids; + DROP TABLE IF EXISTS embeddings; + CREATE TABLE embeddings ( + chunk_id INTEGER PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE, + vector BLOB NOT NULL + ); + CREATE TABLE note_centroids ( + note_b2id TEXT PRIMARY KEY REFERENCES notes(b2id) ON DELETE CASCADE, + centroid BLOB NOT NULL + );", + )?; upsert_meta(conn, "embed_model_id", model_id)?; upsert_meta(conn, "embed_dim", &dim.to_string())?; Ok(()) } /// The `(embed_model_id, embed_dim)` a prior ingest recorded in `meta`, if any. -/// `None` means the vault has never been embedded (no `chunks_vec` yet). This is +/// `None` means the vault has never been embedded (no vector tables yet). This is /// the only place a model swap is detectable, so a read compares it to the active /// embedder and fails fast on a mismatch (index-engine.md §8). pub fn recorded_embedder(conn: &Connection) -> Result> { @@ -452,20 +461,12 @@ fn upsert_meta(conn: &Connection, key: &str, value: &str) -> Result<()> { /// insert never conflicts). pub fn set_chunk_vector(conn: &Connection, chunk_id: i64, embedding: &[f32]) -> Result<()> { conn.execute( - "INSERT INTO chunks_vec(chunk_id, embedding) VALUES (?1, ?2)", + "INSERT INTO embeddings(chunk_id, vector) VALUES (?1, ?2)", params![chunk_id, pack_f32(embedding)], )?; Ok(()) } -/// A note's chunk ids in `seq` order — the note's own vectors/text, e.g. the queries -/// discovery candidate generation runs from (each chunk KNN-searches `chunks_vec`). -pub fn chunks_for_note(conn: &Connection, note_b2id: &str) -> Result> { - let mut stmt = conn.prepare("SELECT id FROM chunks WHERE note_b2id = ?1 ORDER BY seq")?; - let rows = stmt.query_map([note_b2id], |r| r.get(0))?; - Ok(rows.collect::>>()?) -} - /// The note a chunk belongs to (the search-hit → note resolution). pub fn note_for_chunk(conn: &Connection, chunk_id: i64) -> Result> { Ok(conn @@ -478,11 +479,11 @@ pub fn note_for_chunk(conn: &Connection, chunk_id: i64) -> Result } /// The whole `chunk_id → note_b2id` map in one scan — the bulk form of -/// [`note_for_chunk`] for hot loops that resolve *many* hits to their notes. The -/// full-space discovery scan (`discover::candidates`) visits every vault chunk for -/// each anchor chunk, so a per-hit `note_for_chunk` there is an O(anchor × vault) -/// query storm (~463k round-trips, a ~130s `b2 similar`, on a real vault); one map -/// load turns the inner loop into a pointer chase. +/// [`note_for_chunk`] for hot loops that resolve *many* hits to their notes +/// (graph-filtered search walks the full ranked space; a per-hit `note_for_chunk` +/// there is an O(vault) round-trip storm in the worst case — the same N+1 shape +/// that once made `b2 similar` a ~130s stall, #37). One map load turns the inner +/// loop into a pointer chase. pub fn chunk_note_map(conn: &Connection) -> Result> { let mut stmt = conn.prepare("SELECT id, note_b2id FROM chunks")?; let rows = stmt.query_map([], |r| Ok((r.get::<_, i64>(0)?, r.get::<_, String>(1)?)))?; @@ -516,13 +517,14 @@ pub fn note_body_hash(conn: &Connection, b2id: &str) -> Result> { } /// Whether every chunk of `b2id` already has a stored vector (and it has at least -/// one chunk). False after a model swap emptied `chunks_vec`, so an unchanged-body -/// note is still re-embedded then. Requires `chunks_vec` to exist — callers ensure -/// the embedding space first. +/// one chunk). False after a model swap emptied the vector tables, so an +/// unchanged-body note is still re-embedded then. Requires the embedding space to +/// exist — callers ensure it first. A plain indexed anti-join — the vec0 version +/// paid a virtual-table shadow probe per chunk here (#36). pub fn note_fully_embedded(conn: &Connection, b2id: &str) -> Result { let (n_chunks, n_missing): (i64, i64) = conn.query_row( "SELECT COUNT(*), COUNT(*) FILTER (WHERE v.chunk_id IS NULL) - FROM chunks c LEFT JOIN chunks_vec v ON v.chunk_id = c.id + FROM chunks c LEFT JOIN embeddings v ON v.chunk_id = c.id WHERE c.note_b2id = ?1", [b2id], |r| Ok((r.get(0)?, r.get(1)?)), @@ -536,14 +538,14 @@ pub fn note_fully_embedded(conn: &Connection, b2id: &str) -> Result { /// from embedding: nothing is handed between the two passes in memory, so any stop /// point (a cancelled embed, a crash between the passes) heals on the next embed. /// The ordering reproduces the fused reindex's per-note batching + progress. -/// Generalizes [`note_fully_embedded`]; like it, requires `chunks_vec` to exist — -/// callers ensure the embedding space first. +/// Generalizes [`note_fully_embedded`]; like it, requires the embedding space to +/// exist — callers ensure it first. pub fn chunks_missing_vectors(conn: &Connection) -> Result> { let mut stmt = conn.prepare( "SELECT c.note_b2id, n.path, c.id, c.text FROM chunks c JOIN notes n ON n.b2id = c.note_b2id - LEFT JOIN chunks_vec v ON v.chunk_id = c.id + LEFT JOIN embeddings v ON v.chunk_id = c.id WHERE v.chunk_id IS NULL ORDER BY n.path, c.seq", )?; @@ -577,58 +579,93 @@ pub fn all_notes(conn: &Connection) -> Result, _>>()?) } -/// A chunk's stored embedding, unpacked from `chunks_vec` (`None` if the chunk has -/// no vector row). Reading a note's own vectors back is what lets discovery KNN from -/// them without re-embedding — passage↔passage, no `embed_query` (tasks.md ①). Call -/// only when the embedding space exists (`embedding_space_exists`), else the read -/// hits a missing table. -pub fn chunk_vector(conn: &Connection, chunk_id: i64) -> Result>> { - let blob: Option> = conn - .query_row( - "SELECT embedding FROM chunks_vec WHERE chunk_id = ?1", - [chunk_id], - |r| r.get(0), - ) - .optional()?; - Ok(blob.map(|b| crate::embed::unpack_f32(&b))) -} - -/// Brute-force nearest-neighbour search over `chunks_vec`: the `k` nearest chunk ids -/// to `query`, with their distances, nearest first (ties broken by `chunk_id` for -/// determinism). A full linear scan with a **computed** distance — deliberately *not* -/// the `vec0` KNN (`… MATCH … LIMIT k`) operator, whose `k` is hard-capped at 4096 -/// ("k value in knn query too large") — so any `k` is honoured exactly, no silent -/// truncation. This is the brute-force KNN index-engine.md §4 specs as comfortable at -/// vault scale (SQLite keeps only the top `k` for `ORDER BY … LIMIT`). Distance is L2 -/// over the embeddings, which ranks by cosine — b2-embed L2-normalizes, and -/// `chunks_vec` declares no explicit metric so `vec_distance_l2` matches its default. -/// [`vector_search_all`] is the same scan without the `k` bound. -pub fn vector_search(conn: &Connection, query: &[f32], k: usize) -> Result> { - let mut stmt = conn.prepare( - "SELECT chunk_id, vec_distance_l2(embedding, ?1) AS distance FROM chunks_vec - ORDER BY distance, chunk_id LIMIT ?2", +/// A note's stored chunk vectors as `(chunk_id, vector)` in `seq` order — one +/// indexed join, not a per-chunk round-trip. Reading a note's own vectors back is +/// what lets discovery search from them without re-embedding — passage↔passage, no +/// `embed_query` (tasks.md ①); it is also discovery's second-stage rescore unit and +/// the input to a centroid refresh. Call only when the embedding space exists +/// (`embedding_space_exists`), else the read hits a missing table. `prepare_cached` +/// because discovery calls this once per shortlisted note. +pub fn note_chunk_vectors(conn: &Connection, note_b2id: &str) -> Result)>> { + let mut stmt = conn.prepare_cached( + "SELECT c.id, e.vector FROM chunks c + JOIN embeddings e ON e.chunk_id = c.id + WHERE c.note_b2id = ?1 ORDER BY c.seq", )?; - let rows = stmt.query_map(params![pack_f32(query), k as i64], |r| { - Ok((r.get::<_, i64>(0)?, r.get::<_, f64>(1)? as f32)) + let rows = stmt.query_map([note_b2id], |r| { + Ok(( + r.get::<_, i64>(0)?, + crate::embed::unpack_f32(&r.get::<_, Vec>(1)?), + )) })?; Ok(rows.collect::>>()?) } -/// Stream every stored `(chunk_id, embedding_blob)` through `f`, one row at a time — -/// a single full scan of `chunks_vec` that never materializes the whole vector space -/// at once. Discovery's whole-space max-sim (`discover::candidates`) scores every chunk -/// against the anchor's vectors in this one pass, instead of the former one full -/// [`vector_search_all`] scan **per anchor chunk** (which reread — and re-sorted — the -/// entire ~100k-vector space once per anchor chunk: O(anchor × vault) blob reads, the -/// dominant cost of a slow `b2 similar`). The blob is *borrowed* for the callback -/// (`get_ref`), so scoring it adds no per-row allocation. +/// Recompute and store `note_b2id`'s centroid from its currently stored chunk +/// vectors (the row is deleted when it has none). The embed pass calls this after +/// finishing a note, so a centroid row exists exactly for embedded notes and always +/// summarizes their *current* vectors — the derived-projection discipline, no +/// separate invalidation. Requires the embedding space to exist. +pub fn refresh_note_centroid(conn: &Connection, note_b2id: &str) -> Result<()> { + let vectors: Vec> = note_chunk_vectors(conn, note_b2id)? + .into_iter() + .map(|(_, v)| v) + .collect(); + match crate::embed::centroid_of(&vectors) { + Some(c) => { + conn.execute( + "INSERT INTO note_centroids(note_b2id, centroid) VALUES (?1, ?2) + ON CONFLICT(note_b2id) DO UPDATE SET centroid = excluded.centroid", + params![note_b2id, pack_f32(&c)], + )?; + } + None => { + conn.execute( + "DELETE FROM note_centroids WHERE note_b2id = ?1", + [note_b2id], + )?; + } + } + Ok(()) +} + +/// Stream every stored `(note_b2id, centroid_blob)` through `f`, one row at a time — +/// discovery's first-stage coarse scan. O(notes), the whole point of the two-stage +/// shape (#38): the O(chunks) work happens only for the shortlisted notes. The blob +/// is *borrowed* for the callback (`get_ref`), so scoring adds no per-row allocation. +pub fn for_each_note_centroid(conn: &Connection, mut f: impl FnMut(&str, &[u8])) -> Result<()> { + let mut stmt = conn.prepare("SELECT note_b2id, centroid FROM note_centroids")?; + let mut rows = stmt.query([])?; + while let Some(row) = rows.next()? { + // Match the ValueRefs rather than `.as_str()?`/`.as_blob()?` — their + // `FromSqlError` isn't in our error enum, and the column types are fixed by + // our own DDL, so a mismatched row is skipped rather than an error. + let rusqlite::types::ValueRef::Text(text) = row.get_ref(0)? else { + continue; + }; + let Ok(note) = std::str::from_utf8(text) else { + continue; + }; + if let rusqlite::types::ValueRef::Blob(blob) = row.get_ref(1)? { + f(note, blob); + } + } + Ok(()) +} + +/// Stream every stored `(chunk_id, vector_blob)` through `f`, one row at a time — a +/// single sequential scan of the plain `embeddings` table that never materializes +/// the whole vector space at once. One SQL statement for the whole space: the vec0 +/// version of this scan cost a shadow-table probe per row (~38.6k internal +/// statements — and O(vault) log lines — per call on a real vault, #38). The blob is +/// *borrowed* for the callback (`get_ref`), so scoring it adds no per-row allocation. pub fn for_each_stored_vector(conn: &Connection, mut f: impl FnMut(i64, &[u8])) -> Result<()> { - let mut stmt = conn.prepare("SELECT chunk_id, embedding FROM chunks_vec")?; + let mut stmt = conn.prepare("SELECT chunk_id, vector FROM embeddings")?; let mut rows = stmt.query([])?; while let Some(row) = rows.next()? { let chunk_id: i64 = row.get(0)?; // Match the ValueRef rather than `as_blob()?` — its `FromSqlError` isn't in our - // error enum, and a vec0 embedding is always a Blob, so a non-blob is skipped. + // error enum, and a stored vector is always a Blob, so a non-blob is skipped. if let rusqlite::types::ValueRef::Blob(blob) = row.get_ref(1)? { f(chunk_id, blob); } @@ -636,19 +673,46 @@ pub fn for_each_stored_vector(conn: &Connection, mut f: impl FnMut(i64, &[u8])) Ok(()) } +/// Every chunk's squared-L2 distance to `query`, sorted nearest first (ties broken +/// by `chunk_id` for determinism) — the shared scan behind [`vector_search`] / +/// [`vector_search_all`]. Distances are computed **in-process** over the +/// [`for_each_stored_vector`] stream: one sequential statement, one reused decode +/// buffer, the unrolled [`l2_sq`](crate::embed::l2_sq) — the #38 read-path shape. +fn scan_vector_distances(conn: &Connection, query: &[f32]) -> Result> { + let mut out: Vec<(i64, f32)> = Vec::new(); + let mut scratch: Vec = Vec::new(); + for_each_stored_vector(conn, |chunk_id, blob| { + crate::embed::unpack_f32_into(blob, &mut scratch); + out.push((chunk_id, crate::embed::l2_sq(query, &scratch))); + })?; + out.sort_by(|a, b| { + a.1.partial_cmp(&b.1) + .unwrap_or(std::cmp::Ordering::Equal) + .then(a.0.cmp(&b.0)) + }); + Ok(out) +} + +/// Brute-force nearest-neighbour search: the `k` nearest chunk ids to `query`, with +/// their L2 distances, nearest first (ties broken by `chunk_id` for determinism). +/// A full linear scan — exact, no silent truncation at any `k` — which is the +/// brute force index-engine.md §4 specs as comfortable at vault scale. L2 over the +/// stored embeddings ranks by cosine (b2-embed L2-normalizes). The `sqrt` is applied +/// once per *returned* hit; ranking happens on the squared distance (monotonic). +/// [`vector_search_all`] is the same scan without the `k` bound. +pub fn vector_search(conn: &Connection, query: &[f32], k: usize) -> Result> { + let mut hits = scan_vector_distances(conn, query)?; + hits.truncate(k); + Ok(hits.into_iter().map(|(id, d)| (id, d.sqrt())).collect()) +} + /// [`vector_search`] without the `k` bound: **every** chunk's distance to `query`, -/// nearest first (same computed-L2 scan, same `chunk_id` tie-break). The whole-space -/// caller — graph-filtered search — ranks the entire vault, so it takes this rather -/// than pass a sentinel `k`. +/// nearest first (same scan, same `chunk_id` tie-break). The whole-space caller — +/// graph-filtered search — ranks the entire vault, so it takes this rather than +/// pass a sentinel `k`. pub fn vector_search_all(conn: &Connection, query: &[f32]) -> Result> { - let mut stmt = conn.prepare( - "SELECT chunk_id, vec_distance_l2(embedding, ?1) AS distance FROM chunks_vec - ORDER BY distance, chunk_id", - )?; - let rows = stmt.query_map(params![pack_f32(query)], |r| { - Ok((r.get::<_, i64>(0)?, r.get::<_, f64>(1)? as f32)) - })?; - Ok(rows.collect::>>()?) + let hits = scan_vector_distances(conn, query)?; + Ok(hits.into_iter().map(|(id, d)| (id, d.sqrt())).collect()) } // --------------------------------------------------------------------------- diff --git a/crates/b2-core/src/discover.rs b/crates/b2-core/src/discover.rs index ee5d153..04482ba 100644 --- a/crates/b2-core/src/discover.rs +++ b/crates/b2-core/src/discover.rs @@ -10,31 +10,52 @@ //! a scoped-traversal primitive, the wrong tool here.) Generation is deliberately //! **permissive**: it over-produces, and the human decides which are worth a link. //! -//! Mechanics (tasks.md ①): score every stored chunk by its similarity to the anchor's -//! **nearest** stored chunk vector, keep each note's **best** such chunk (max-sim), and -//! subtract the anchor and its **direct (1-hop)** neighbors. This is one whole-space -//! pass over `chunks_vec` (`db::for_each_stored_vector`), computing squared-L2 in -//! process — *not* one SQL KNN scan per anchor chunk, which reread and re-sorted the -//! entire vector space once per anchor chunk (O(anchor × vault), the old `b2 similar` -//! stall). It is vector-only and **re-embeds nothing** — discovery is passage↔passage, -//! so the anchor is represented by the vectors already in `chunks_vec`, never by an -//! `embed_query` of its text (bge's asymmetric query prefix is the wrong side). -//! Graph distance beyond the 1-hop exclusion is **not** a ranking signal — graph- -//! distant "bridge" candidates ride along unboosted; weighting distance (closure vs. -//! serendipity) is a deferred, eval-gated experiment (tasks.md backlog). +//! Mechanics are **two-stage** (#38; planning/research/discovery-scan-strategy.md): +//! +//! 1. **Coarse, O(notes):** rank every note by the distance of its stored *centroid* +//! (`note_centroids`, maintained by the embed pass) to the anchor's centroid, +//! minus the anchor and its direct (1-hop) neighbors, and keep a shortlist many +//! times larger than `limit`. +//! 2. **Exact, O(shortlist):** for each shortlisted note, load its chunk vectors and +//! score the exact max-sim — the best pair across the anchor's chunks × that +//! note's chunks — keeping the chunk that achieved it as evidence. +//! +//! Only the shortlist changes with stage 1; stage 2's scoring is the same exact +//! max-sim the previous whole-space scan computed, so a shortlist that covers the +//! vault (any small/test vault) reproduces it exactly. What the shape buys: the +//! per-open heavy pass reads N_notes centroid rows instead of N_chunks vector rows — +//! effectively flat as the vault grows (the previous exact scan was ~4.4 s at ~38.6k +//! chunks, #38). Discovery is vector-only and **re-embeds nothing** — the anchor is +//! represented by the vectors already stored, never by an `embed_query` of its text +//! (bge's asymmetric query prefix is the wrong side). Graph distance beyond the +//! 1-hop exclusion is **not** a ranking signal — graph-distant "bridge" candidates +//! ride along unboosted; weighting distance (closure vs. serendipity) is a deferred, +//! eval-gated experiment (tasks.md backlog). use crate::db; -use crate::embed::{l2_sq, unpack_f32}; +use crate::embed::{centroid_of, l2_sq, unpack_f32_into}; use crate::error::Result; use crate::graph; use rusqlite::Connection; -use std::collections::HashMap; /// The exclusion radius: a candidate must not already be *directly* linked to the /// anchor. Fixed at 1 by decision (tasks.md ①) so triadic-closure candidates — a note /// two hops away, transitively related but with no direct edge — stay in the pool. const EXCLUDE_HOPS: usize = 1; +/// Floor on the stage-1 shortlist. Generous relative to any `limit` a human-facing +/// surface asks for: discovery is recall-oriented (the human is the precision gate), +/// so the coarse stage must never be the reason a nearby note goes missing. On any +/// vault at or below this many candidate notes the two-stage result is *exactly* +/// the old whole-space scan's. +const SHORTLIST_MIN: usize = 200; + +/// Stage-1 shortlist size per requested result: `limit × this`, floored at +/// [`SHORTLIST_MIN`]. A wide margin over `limit` because a note's centroid can rank +/// a few places below where its single best chunk deserves (the centroid smooths +/// over the note's chunks); the exact stage re-ranks whatever survives. +const SHORTLIST_PER_RESULT: usize = 20; + /// One discovery candidate: a note near the anchor and not already connected, ranked /// by `score`. Owned, so the façade can resolve it to a [`SimilarView`](crate::vault::SimilarView) /// for `b2 similar` without threading a lifetime through generation. @@ -43,8 +64,7 @@ pub struct CandidateNote { /// The candidate note's `b2id`. pub note_b2id: String, /// Best chunk-pair similarity across the anchor's chunks × this note's chunks — - /// higher is nearer (negated `sqlite-vec` distance, matching - /// [`Hit`](crate::search::Hit)). + /// higher is nearer (negated L2 distance, matching [`Hit`](crate::search::Hit)). pub score: f64, /// The candidate's chunk that achieved `score` — the passage that made this note /// similar, surfaced by `b2 similar` as the evidence for *why* it appeared. @@ -55,83 +75,76 @@ pub struct CandidateNote { /// first (ties broken by `note_b2id` for determinism). /// /// Returns empty when the vault has no embedding space yet, when the anchor has no -/// chunks (unknown or empty note), or when `limit` is 0 — there is nothing to search -/// from. Excludes the anchor itself and its direct neighbors; everything else near in -/// vector space is a candidate. +/// stored vectors (unknown, empty, or not-yet-embedded note), or when `limit` is 0 — +/// there is nothing to search from. Excludes the anchor itself and its direct +/// neighbors; everything else near in vector space is a candidate. pub fn candidates(conn: &Connection, anchor: &str, limit: usize) -> Result> { if limit == 0 || !db::embedding_space_exists(conn)? { return Ok(Vec::new()); } - let anchor_chunks = db::chunks_for_note(conn, anchor)?; - if anchor_chunks.is_empty() { + // The anchor's own stored vectors, loaded once (re-embeds nothing — tasks.md ①); + // none ⇒ nothing to search from. Its centroid is computed in-process from them + // rather than read back, so an anchor mid-embed still discovers from what it has. + let anchor_vecs: Vec> = db::note_chunk_vectors(conn, anchor)? + .into_iter() + .map(|(_, v)| v) + .collect(); + let Some(anchor_centroid) = centroid_of(&anchor_vecs) else { return Ok(Vec::new()); - } + }; // The only use of the graph in generation: subtract what's already linked — the // anchor and everything within 1 hop (self + direct neighbors). let exclude = graph::reachable_within(conn, anchor, EXCLUDE_HOPS)?; - // The anchor's own stored vectors (re-embeds nothing — tasks.md ①), unpacked once - // so the hot scan below reuses them across every stored chunk. A chunk with no - // stored vector (shouldn't occur post-embed) is skipped; none at all ⇒ nothing to - // search from. - let mut anchor_vecs: Vec> = Vec::new(); - for chunk in anchor_chunks { - if let Some(v) = db::chunk_vector(conn, chunk)? { - anchor_vecs.push(v); - } - } - if anchor_vecs.is_empty() { - return Ok(Vec::new()); - } - - // Resolve chunk→note once, up front — the scan visits every vault chunk, so a - // per-hit note_for_chunk query would be an O(vault) round-trip storm. - let chunk_note = db::chunk_note_map(conn)?; - - // One whole-space pass (not one KNN scan per anchor chunk): score every stored - // chunk by its nearest anchor vector and keep, per note, its best chunk — exact - // max-sim, the brute force index-engine.md §4 specs as comfortable at vault scale. - // `best`: note_b2id → (smallest squared L2 seen, the chunk that achieved it). - // Squared L2 is the same ranking key as `vec_distance_l2` without the per-hit - // `sqrt` (monotonic); the `sqrt` is applied once per candidate below. - let mut best: HashMap = HashMap::new(); - db::for_each_stored_vector(conn, |chunk_id, blob| { - let Some(note) = chunk_note.get(&chunk_id) else { - return; - }; - if exclude.contains(note.as_str()) { + // Stage 1 — coarse shortlist over note centroids: one O(notes) scan, excluded + // notes skipped up front so they never occupy a shortlist slot. + let mut coarse: Vec<(f32, String)> = Vec::new(); + let mut scratch: Vec = Vec::new(); + db::for_each_note_centroid(conn, |note, blob| { + if exclude.contains(note) { return; // the anchor or a direct neighbor — already connected } - // Decode this stored chunk once, then take its nearest anchor vector (min over - // the anchor's chunks) — max-sim's inner min. - let v = unpack_f32(blob); - let Some(dist_sq) = anchor_vecs - .iter() - .map(|a| l2_sq(a, &v)) - .min_by(|x, y| x.partial_cmp(y).unwrap_or(std::cmp::Ordering::Equal)) - else { - return; - }; - // Look up before inserting so the note id is cloned only on first sighting - // (≤ once per distinct note), not on every one of the vault-many hits. - match best.get_mut(note) { - Some(cur) if dist_sq < cur.0 => *cur = (dist_sq, chunk_id), - Some(_) => {} - None => { - best.insert(note.clone(), (dist_sq, chunk_id)); + unpack_f32_into(blob, &mut scratch); + coarse.push((l2_sq(&anchor_centroid, &scratch), note.to_string())); + })?; + coarse.sort_by(|a, b| { + a.0.partial_cmp(&b.0) + .unwrap_or(std::cmp::Ordering::Equal) + .then(a.1.cmp(&b.1)) + }); + coarse.truncate( + limit + .saturating_mul(SHORTLIST_PER_RESULT) + .max(SHORTLIST_MIN), + ); + + // Stage 2 — exact max-sim over the shortlist only: per note, the best (smallest + // squared-L2) pair across the anchor's chunks × its chunks. Squared L2 is the + // same ranking key as L2 without the per-comparison `sqrt` (monotonic); the + // `sqrt` is applied once per surfaced candidate below. Strictly-less keeps the + // earliest (lowest-`seq`) chunk on ties, deterministically. A shortlisted note + // with no stored chunk vectors (possible mid-embed) scores nothing and drops out. + let mut out: Vec = Vec::new(); + for (_, note_b2id) in coarse { + let mut best: Option<(f32, i64)> = None; + for (chunk_id, v) in db::note_chunk_vectors(conn, ¬e_b2id)? { + for a in &anchor_vecs { + let dist_sq = l2_sq(a, &v); + if best.is_none_or(|(cur, _)| dist_sq < cur) { + best = Some((dist_sq, chunk_id)); + } } } - })?; + if let Some((dist_sq, evidence_chunk_id)) = best { + out.push(CandidateNote { + note_b2id, + score: -(dist_sq.sqrt() as f64), // nearer = higher, matching Hit's -L2 + evidence_chunk_id, + }); + } + } - let mut out: Vec = best - .into_iter() - .map(|(note_b2id, (dist_sq, evidence_chunk_id))| CandidateNote { - note_b2id, - score: -(dist_sq.sqrt() as f64), // nearer = higher, matching Hit's -L2 - evidence_chunk_id, - }) - .collect(); // Best score first; ties broken by id so the ranking (and thus `limit`'s prefix) // is deterministic. out.sort_by(|a, b| { diff --git a/crates/b2-core/src/embed.rs b/crates/b2-core/src/embed.rs index 9165b60..78afd13 100644 --- a/crates/b2-core/src/embed.rs +++ b/crates/b2-core/src/embed.rs @@ -7,8 +7,8 @@ use crate::error::Result; -/// Turns note text into a vector. The dimension is fixed per model and pins the -/// `chunks_vec` column type via `meta.embed_dim` (build spec §1.0/§1.2). +/// Turns note text into a vector. The dimension is fixed per model and recorded +/// as `meta.embed_dim` (build spec §1.0/§1.2). /// /// `embed` is **fallible**: the fake never fails, but a real model runs tensor /// math that can (e.g. a device/allocation error), and the index path must surface @@ -19,9 +19,9 @@ use crate::error::Result; /// override `embed_query`; the default is symmetric. pub trait Embedder { /// Stable identifier recorded in `meta.embed_model_id`. A change to it (or to - /// `dim`) is a model swap → drop `chunks_vec` + re-embed (index-engine.md §8). + /// `dim`) is a model swap → drop the stored vectors + re-embed (index-engine.md §8). fn model_id(&self) -> &str; - /// Vector dimension; must equal the `FLOAT[N]` literal of `chunks_vec`. + /// Vector dimension; must equal the recorded `meta.embed_dim`. fn dim(&self) -> usize; /// Embed one document/passage for indexing. fn embed(&self, text: &str) -> Result>; @@ -91,9 +91,9 @@ impl Embedder for FakeEmbedder { } } -/// Pack a vector as the compact little-endian float32 BLOB that `sqlite-vec` -/// accepts for `vec0` columns (build spec §1.2). The query side packs the same -/// way so an exact match has distance 0. +/// Pack a vector as a compact little-endian float32 BLOB — the stored form of every +/// vector in the index (`embeddings.vector`, `note_centroids.centroid`; build spec +/// §1.2). The query side packs the same way so an exact match has distance 0. pub fn pack_f32(v: &[f32]) -> Vec { let mut out = Vec::with_capacity(v.len() * 4); for x in v { @@ -102,11 +102,11 @@ pub fn pack_f32(v: &[f32]) -> Vec { out } -/// Inverse of [`pack_f32`]: read a `sqlite-vec` `vec0` BLOB (little-endian float32, -/// no header) back into a vector. Used to reuse a note's *stored* chunk vectors as -/// KNN queries without re-embedding (connection-discovery candidate generation, -/// tasks.md ①). A trailing partial group can't occur for a `FLOAT[N]` column, so a -/// non-multiple-of-4 length is simply truncated rather than treated as an error. +/// Inverse of [`pack_f32`]: read a stored BLOB (little-endian float32, no header) +/// back into a vector. Used to reuse a note's *stored* chunk vectors as discovery +/// queries without re-embedding (tasks.md ①). A trailing partial group can't occur +/// for a vector written by [`pack_f32`], so a non-multiple-of-4 length is simply +/// truncated rather than treated as an error. pub fn unpack_f32(bytes: &[u8]) -> Vec { bytes .chunks_exact(4) @@ -114,19 +114,76 @@ pub fn unpack_f32(bytes: &[u8]) -> Vec { .collect() } -/// Squared Euclidean distance between two equal-length vectors. This is the ranking -/// key `sqlite-vec`'s `vec_distance_l2` sorts by, minus the final `sqrt` — `sqrt` is -/// monotonic, so dropping it never changes an ordering. Scoring off the squared -/// distance (and applying `sqrt` once per surfaced candidate, not per comparison) is -/// what lets discovery rank the *whole* vector space in one in-process pass instead of -/// one SQL KNN scan per anchor chunk. A length mismatch (impossible for a fixed -/// `FLOAT[N]` column) scores the shared prefix rather than panicking. +/// [`unpack_f32`] into a caller-owned scratch buffer, reusing its capacity. The +/// whole-space scans decode one stored vector per visited row; a fresh `Vec` per row +/// was a measurable slice of the `b2 similar` stall (#38), where this costs nothing +/// after the first row. +pub fn unpack_f32_into(bytes: &[u8], out: &mut Vec) { + out.clear(); + out.extend( + bytes + .chunks_exact(4) + .map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]])), + ); +} + +/// Squared Euclidean distance between two equal-length vectors — the index's one +/// ranking key, minus the final `sqrt` (`sqrt` is monotonic, so dropping it never +/// changes an ordering; it is applied once per *surfaced* result, not per +/// comparison). A length mismatch (impossible for vectors from one embedding space) +/// scores the shared prefix rather than panicking. +/// +/// Eight independent accumulators, summed at the end: float addition is +/// non-associative, so a single running sum forms one serial dependency chain the +/// compiler must execute as written — splitting it lets LLVM autovectorize. +/// Measured at the #38 scale (38.6k × 768-dim, 12 anchors) the naive iterator shape +/// cost ~530 ms; this shape ~75 ms. pub fn l2_sq(a: &[f32], b: &[f32]) -> f32 { - a.iter() - .zip(b) - .map(|(x, y)| { - let d = x - y; - d * d - }) - .sum() + let n = a.len().min(b.len()); + let (a, b) = (&a[..n], &b[..n]); + let mut acc = [0.0f32; 8]; + let chunks_a = a.chunks_exact(8); + let chunks_b = b.chunks_exact(8); + let (tail_a, tail_b) = (chunks_a.remainder(), chunks_b.remainder()); + for (xa, xb) in chunks_a.zip(chunks_b) { + for i in 0..8 { + let d = xa[i] - xb[i]; + acc[i] += d * d; + } + } + let mut sum: f32 = acc.iter().sum(); + for (x, y) in tail_a.iter().zip(tail_b) { + let d = x - y; + sum += d * d; + } + sum +} + +/// The **centroid** of a note's chunk vectors: their arithmetic mean, L2-normalized +/// (the spherical mean — the standard coarse representative when the underlying +/// vectors are cosine-normalized, as b2-embed's are). `None` for an empty set — a +/// note with no stored vectors has no centroid row. Deterministic: summation runs in +/// the given (chunk `seq`) order. Discovery's first stage ranks whole *notes* by +/// centroid distance, so its heavy scan is O(notes), not O(chunks) (#38). +pub fn centroid_of(vectors: &[Vec]) -> Option> { + let first = vectors.first()?; + let mut mean = vec![0.0f32; first.len()]; + for v in vectors { + // A length mismatch can't occur within one embedding space; fold the shared + // prefix rather than panic, mirroring `l2_sq`. + for (m, x) in mean.iter_mut().zip(v) { + *m += x; + } + } + let n = vectors.len() as f32; + for m in &mut mean { + *m /= n; + } + let norm = mean.iter().map(|x| x * x).sum::().sqrt(); + if norm > 0.0 { + for m in &mut mean { + *m /= norm; + } + } + Some(mean) } diff --git a/crates/b2-core/src/ingest.rs b/crates/b2-core/src/ingest.rs index 7338829..1c5d17c 100644 --- a/crates/b2-core/src/ingest.rs +++ b/crates/b2-core/src/ingest.rs @@ -111,8 +111,8 @@ pub struct ReindexProgress { /// `consult_vectors` selects the re-chunk predicate. The full-vault projection pass /// passes `false`: it reads only `notes` (`force || body changed || note is new`), /// because "unchanged body but missing vectors" is [`embed_vault`]'s job, not a -/// reason to re-chunk — and this is what keeps [`project_vault`] free of -/// `chunks_vec` (projection-embedding-split.md §4). [`ingest_file`] passes `true` +/// reason to re-chunk — and this is what keeps [`project_vault`] free of the +/// vector tables (projection-embedding-split.md §4). [`ingest_file`] passes `true` /// (it embeds inline and has ensured the space exists), so a note left mid-embed is /// also healed by [`would_reembed`]'s vector-state check, exactly as before. fn project_note_and_chunks( @@ -181,7 +181,7 @@ fn project_note_and_chunks( // Incremental fast path: an unchanged body means identical chunks — reuse them // and return no pending work (`rechunk = false`). `force` bypasses this; on the - // inline path so does a model swap, which emptied `chunks_vec` + // inline path so does a model swap, which emptied the vector tables // (note_fully_embedded then returns false). let pending = if rechunk { // Chunk → project rows; hand the (id, text) pairs back for a batched embed @@ -203,13 +203,13 @@ fn project_note_and_chunks( /// incremental "unchanged" fast path: true when `force`, when the vault has no /// embedding space yet (`space_exists = false` → a pristine/never-embedded index), /// when the stored body hash differs (content changed), or when the note is not -/// fully embedded (a fresh note, or a model swap emptied `chunks_vec`). Shared by -/// the inline single-note ingest ([`ingest_file`]) and the [`plan_reindex`] dry-run. +/// fully embedded (a fresh note, or a model swap emptied the vector tables). Shared +/// by the inline single-note ingest ([`ingest_file`]) and the [`plan_reindex`] dry-run. /// [`project_vault`] deliberately does **not** use it (projection never reads vector /// state); the dry-run's `would_embed` still predicts the composed project+embed run /// correctly, since a body-changed *or* vector-missing note both end up embedded. -/// `space_exists` lets a pristine vault short-circuit without querying a -/// `chunks_vec` that does not exist yet (which would error). +/// `space_exists` lets a pristine vault short-circuit without querying an +/// `embeddings` table that does not exist yet (which would error). fn would_reembed( conn: &Connection, b2id: &str, @@ -237,7 +237,7 @@ struct NoteEmbedOutcome { completed: bool, } -/// Embed a note's pending `(chunk_id, text)` pairs into `chunks_vec`, in batches of +/// Embed a note's pending `(chunk_id, text)` pairs into `embeddings`, in batches of /// [`EMBED_BATCH`] via [`Embedder::embed_batch`], calling `on_batch` with each /// batch's size (so a full reindex can report cumulative progress **and** cooperatively /// cancel). Chunk vectors are independent, so batch boundaries never change the result. @@ -245,14 +245,25 @@ struct NoteEmbedOutcome { /// The cancel check runs **after** a batch is fully written, so a cancel never tears a /// batch (async-indexing.md §5.6) — it only stops *further* batches. Returns whether a /// cancel was seen and whether the note finished embedding (see [`NoteEmbedOutcome`]). +/// +/// A note that finishes has its **centroid** refreshed from its now-complete stored +/// vectors (`note_centroids` — discovery's coarse stage, #38): the centroid is +/// derived data with the same lifecycle as the vectors themselves, so maintaining it +/// here — the one place vectors are written — means no other pass ever reconciles +/// it. A note cut off mid-embed skips the refresh; its resume completes the vectors +/// and refreshes then. Running this even when `pending` is empty is deliberate: it +/// costs one indexed read and re-derives (or heals a missing) centroid for an +/// already-embedded note. fn embed_pending( conn: &Connection, embedder: &dyn Embedder, + note_b2id: &str, pending: &[(i64, String)], mut on_batch: impl FnMut(usize) -> ControlFlow<()>, ) -> Result { let total = pending.len(); let mut done = 0usize; + let mut cancelled = false; for batch in pending.chunks(EMBED_BATCH) { let texts: Vec<&str> = batch.iter().map(|(_, t)| t.as_str()).collect(); let vectors = embedder.embed_batch(&texts)?; @@ -261,15 +272,17 @@ fn embed_pending( } done += batch.len(); if on_batch(batch.len()).is_break() { - return Ok(NoteEmbedOutcome { - cancelled: true, - completed: done == total, - }); + cancelled = true; + break; } } + let completed = done == total; + if completed { + db::refresh_note_centroid(conn, note_b2id)?; + } Ok(NoteEmbedOutcome { - cancelled: false, - completed: true, + cancelled, + completed, }) } @@ -359,7 +372,9 @@ pub fn ingest_file( project_note_and_chunks(conn, vault_root, rel_path, idgen, false, true)?; let embedded = !pending.is_empty(); // A single-note re-projection is never cancelled — always run to completion. - embed_pending(conn, embedder, &pending, |_| ControlFlow::Continue(()))?; + embed_pending(conn, embedder, &b2id, &pending, |_| { + ControlFlow::Continue(()) + })?; project_edges(conn, &b2id, &body, &relations)?; Ok(Ingested { b2id, @@ -485,8 +500,8 @@ pub struct EmbedOutcome { /// The **projection pass** (projection-embedding-split.md §4): project every `.md` /// file under `vault_root` — Phase 1 (note + chunks + FTS, stamping missing /// `b2id`s) then Phase 2 (the typed edges) — with **no embedder and no embedding -/// space**: it never touches `chunks_vec`, so it needs neither the model nor its -/// `dim`, and a projected-but-unembedded index is already complete for keyword +/// space**: it never creates the vector tables, so it needs neither the model nor +/// its `dim`, and a projected-but-unembedded index is already complete for keyword /// search and the graph. Incremental: unless `force`, a note is re-chunked only /// when its body changed or it is new — read purely from `notes`, never from vector /// state (missing vectors are [`embed_vault`]'s job). Re-chunking a previously @@ -551,9 +566,10 @@ pub fn project_vault( } /// The **embed pass** (projection-embedding-split.md §4): fill a vector for every -/// chunk that lacks one. Ensures the embedding space first (creates `chunks_vec` at -/// the embedder's `dim`; a model swap drops + resets it, so *all* chunks then count -/// as missing), then works the DB-derived pending set ([`db::chunks_missing_vectors`]) +/// chunk that lacks one. Ensures the embedding space first (creates the +/// `embeddings` + `note_centroids` tables; a model swap drops + resets them, so +/// *all* chunks then count as missing), then works the DB-derived pending set +/// ([`db::chunks_missing_vectors`]) /// note by note through the batched [`embed_pending`] loop — firing `on_progress` /// per batch and honoring its [`ControlFlow::Break`] as the cooperative cancel /// checkpoint (async-indexing.md §3). Takes **no `force`**: re-chunking (which @@ -601,7 +617,7 @@ pub fn embed_vault( .entered(); let notes_embedded = i + 1; // 1-based position for the progress line let note_chunks = pending.len(); - let outcome = embed_pending(conn, embedder, pending, |n| { + let outcome = embed_pending(conn, embedder, b2id, pending, |n| { chunks_done += n; on_progress(ReindexProgress { note_path: path.clone(), diff --git a/crates/b2-core/src/lib.rs b/crates/b2-core/src/lib.rs index ab082ac..33b0a9f 100644 --- a/crates/b2-core/src/lib.rs +++ b/crates/b2-core/src/lib.rs @@ -2,8 +2,9 @@ //! //! Built step 0→5 per `planning/specs/completed/index-engine-build.md`; the schema is a //! derived projection of `planning/data-model.md` and must satisfy it, never the -//! reverse. Step 0 is the substrate: open the DB with the locked pragmas and prove -//! FTS5 (BM25) and `sqlite-vec` (KNN) coexist in one statically-linked connection. +//! reverse. Step 0 is the substrate: open the DB with the locked pragmas over the +//! bundled, statically-linked SQLite (FTS5 compiled in; vectors are plain BLOB +//! tables scored in-process since schema v3, #38 — no extension needed). pub mod add; pub mod chunk; diff --git a/crates/b2-core/src/search.rs b/crates/b2-core/src/search.rs index 72c7250..32c77a0 100644 --- a/crates/b2-core/src/search.rs +++ b/crates/b2-core/src/search.rs @@ -1,8 +1,9 @@ //! Hybrid retrieval (planning/index-engine.md §1, §5; build spec Flow ②). //! -//! BM25 (over `chunks_fts`) and brute-force vector KNN (over `chunks_vec`) are -//! retrieved in parallel and fused with **Reciprocal Rank Fusion** (`Σ 1/(k+rank+1)`, -//! k=60), borrowed wholesale from qmd. Results resolve up from chunks to notes. +//! BM25 (over `chunks_fts`) and brute-force vector KNN (an in-process scan over the +//! stored `embeddings`) are retrieved in parallel and fused with **Reciprocal Rank +//! Fusion** (`Σ 1/(k+rank+1)`, k=60), borrowed wholesale from qmd. Results resolve +//! up from chunks to notes. //! //! The graph-filtered variant is B2's reason to exist: "nearest chunks whose note //! is within k typed hops of note X" — the vector⨝graph join (index-engine.md §3) @@ -93,7 +94,7 @@ fn pool_size(limit: usize) -> usize { /// Keyword-only search: BM25 over `chunks_fts` → top `limit`, resolved to notes — /// the fallback that makes a **projected-but-unembedded** vault searchable -/// (projection-embedding-split.md §5): no query embedding, no model, no `chunks_vec`. +/// (projection-embedding-split.md §5): no query embedding, no model, no vectors. /// Scores are the RRF of the single BM25 list, so they live on the same scale (and /// sort the same way) as [`hybrid_search`]'s fused scores. pub fn keyword_only_search( @@ -160,9 +161,11 @@ pub fn hybrid_search( /// /// Reachability is undirected over `active` edges (a note related to the anchor /// either way is a candidate). Filtering is done by scanning the full ranked space -/// and keeping reachable notes — exact at vault scale (a full brute-force scan); the -/// precise scale lever is a partition column on `note_b2id` for filtered KNN (build -/// spec §1.2 / §4). +/// and keeping reachable notes — exact at vault scale (a full brute-force scan). +/// Chunk→note resolution is one bulk map load, not a per-ranked-row query: the walk +/// visits ranked chunks until `limit` reachable ones are found, which in the worst +/// case (a small neighborhood ranked deep) is the whole vault — the same N+1 shape +/// that once stalled `b2 similar` (#37). pub fn graph_filtered_search( conn: &rusqlite::Connection, embedder: &dyn Embedder, @@ -172,16 +175,17 @@ pub fn graph_filtered_search( limit: usize, ) -> Result> { let reachable = graph::reachable_within(conn, anchor, hops)?; + let chunk_note = db::chunk_note_map(conn)?; let mut hits = Vec::new(); for (chunk_id, distance) in db::vector_search_all(conn, &embedder.embed_query(query)?)? { - let Some(note_b2id) = db::note_for_chunk(conn, chunk_id)? else { + let Some(note_b2id) = chunk_note.get(&chunk_id) else { continue; }; - if reachable.contains(¬e_b2id) { + if reachable.contains(note_b2id) { hits.push(Hit { chunk_id, - note_b2id, + note_b2id: note_b2id.clone(), score: -(distance as f64), // closer = higher }); if hits.len() == limit { diff --git a/crates/b2-core/src/vault.rs b/crates/b2-core/src/vault.rs index 7135438..45ac579 100644 --- a/crates/b2-core/src/vault.rs +++ b/crates/b2-core/src/vault.rs @@ -257,7 +257,7 @@ impl Vault { /// to inject the real candle model while tests keep the fake. /// /// `open` **never mutates the embedding space** (the `open()`-time-drop fix, - /// tasks.md / index-engine.md §8): shaping `chunks_vec` and any re-embed happen + /// tasks.md / index-engine.md §8): shaping the vector tables and any re-embed happen /// only on `reindex`. That way changing the configured model can never silently /// wipe vectors on the next command — a mismatch is caught, and fixed, at /// `reindex`; `search` fails fast on it (see [`search`](Self::search)). @@ -344,7 +344,7 @@ impl Vault { } /// The **embed pass** alone: fill a vector for every chunk that lacks one — the - /// pending set is derived from the index itself (chunks with no `chunks_vec` + /// pending set is derived from the index itself (chunks with no `embeddings` /// row), so this needs no prior [`project`](Self::project) call in the same /// process and heals any interruption (a cancelled embed, a crash between the /// passes) by embedding exactly what is still missing. Progress and cooperative diff --git a/crates/b2-core/tests/cancel.rs b/crates/b2-core/tests/cancel.rs index 45ed799..8696c0a 100644 --- a/crates/b2-core/tests/cancel.rs +++ b/crates/b2-core/tests/cancel.rs @@ -53,7 +53,7 @@ fn cancel_after_first_batch_leaves_a_consistent_resumable_index() { ); // …only VECTORS are partial: a prefix of chunks embedded, the rest pending. - let vecs_after_cancel = count(&conn, "chunks_vec"); + let vecs_after_cancel = count(&conn, "embeddings"); assert!( vecs_after_cancel > 0 && vecs_after_cancel < chunks, "a prefix embedded, the remainder pending: {vecs_after_cancel}/{chunks}" @@ -68,7 +68,7 @@ fn cancel_after_first_batch_leaves_a_consistent_resumable_index() { .unwrap(); assert!(!resumed.cancelled); assert_eq!( - count(&conn, "chunks_vec"), + count(&conn, "embeddings"), chunks, "resume fills the remaining vectors — the index is now fully embedded" ); diff --git a/crates/b2-core/tests/discover.rs b/crates/b2-core/tests/discover.rs index 08ce874..eddd33b 100644 --- a/crates/b2-core/tests/discover.rs +++ b/crates/b2-core/tests/discover.rs @@ -148,6 +148,88 @@ fn a_directly_connected_pair_yields_no_candidates() { .is_empty()); } +/// Two-stage discovery (coarse centroid shortlist → exact rescore, #38) must equal +/// the **exhaustive** whole-space max-sim whenever the shortlist covers the +/// candidate set — which it always does at test scale (the shortlist floor is 200 +/// notes). Ground truth is recomputed here straight from the stored vectors, +/// independent of `discover`'s code path: same max-sim, same tie rules, over every +/// chunk in the vault. With the fake embedder the *ordering* is arbitrary (random +/// vectors), which is exactly why full equality — notes, scores, evidence chunks — +/// is a strong plumbing check. +#[test] +fn two_stage_equals_exhaustive_max_sim_when_shortlist_covers() { + use b2_core::embed::{l2_sq, unpack_f32}; + use std::collections::HashMap; + + const NOTES: usize = 40; + const PARAS: usize = 4; + + let tmp = tempfile::TempDir::new().unwrap(); + let vault = tmp.path().join("vault"); + fs::create_dir_all(&vault).unwrap(); + let mut ids = Vec::new(); + for n in 0..NOTES { + let b2id = format!("01JN{n:022}"); + let body = (0..PARAS) + .map(|p| format!("note {n} para {p}: topic {}", (n * 31 + p * 7) % 97)) + .collect::>() + .join("\n\n"); + write_note(&vault, &format!("n{n}.md"), &b2id, &body); + ids.push(b2id); + } + let conn = open(&tmp.path().join("b2.sqlite")).unwrap(); + ingest_vault(&conn, &vault, &UlidGen, &FakeEmbedder::new(64)).unwrap(); + + let anchor = &ids[0]; + let anchor_vecs: Vec> = db::note_chunk_vectors(&conn, anchor) + .unwrap() + .into_iter() + .map(|(_, v)| v) + .collect(); + + // Exhaustive ground truth: every stored vector, min over the anchor's vectors, + // best chunk per note (strictly-less keeps the first-seen chunk, as discover does). + let chunk_note = db::chunk_note_map(&conn).unwrap(); + let mut best: HashMap = HashMap::new(); + db::for_each_stored_vector(&conn, |chunk_id, blob| { + let note = &chunk_note[&chunk_id]; + if note == anchor { + return; // no links in this vault → the anchor is the whole exclusion set + } + let v = unpack_f32(blob); + for a in &anchor_vecs { + let d = l2_sq(a, &v); + let cur = best.entry(note.clone()).or_insert((f32::INFINITY, 0)); + if d < cur.0 { + *cur = (d, chunk_id); + } + } + }) + .unwrap(); + let mut expected: Vec = best + .into_iter() + .map(|(note_b2id, (d, evidence_chunk_id))| CandidateNote { + note_b2id, + score: -(d.sqrt() as f64), + evidence_chunk_id, + }) + .collect(); + expected.sort_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap() + .then(a.note_b2id.cmp(&b.note_b2id)) + }); + + let got = discover::candidates(&conn, anchor, NOTES).unwrap(); + assert_eq!(got.len(), NOTES - 1, "every other note is a candidate"); + assert_eq!( + got, expected, + "two-stage discovery must reproduce the exhaustive scan exactly \ + (notes, scores, and evidence chunks)" + ); +} + #[test] fn unknown_or_chunkless_anchor_and_zero_limit_yield_no_candidates() { let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/b2-core/tests/discover_query_count.rs b/crates/b2-core/tests/discover_query_count.rs index e6306a9..30d5a80 100644 --- a/crates/b2-core/tests/discover_query_count.rs +++ b/crates/b2-core/tests/discover_query_count.rs @@ -1,10 +1,10 @@ -//! Regression: `discover::candidates` must resolve chunk→note in **one bulk query**, -//! not a per-hit `note_for_chunk` round-trip. The full-space scan visits every vault -//! chunk for *each* anchor chunk, so a per-hit lookup was O(anchor_chunks × vault_chunks) -//! — on a real ~38k-chunk vault a 12-chunk anchor fired ~463k of these, turning -//! `b2 similar` (the desktop's note-open discovery) into a ~130s stall and a 277 MB -//! `B2_LOG` file. This locks the fix: per-hit note resolution must not scale with the -//! product. +//! Regression: `discover::candidates` must not issue O(chunks) SQL — neither a +//! per-hit `note_for_chunk` round-trip (#37's N+1: ~463k statements, a ~130s +//! `b2 similar` on a real vault) nor a whole-space chunk-vector scan per open +//! (#38: exact brute force over every stored vector — ~38.6k rows read, and under +//! the old vec0 store ~38.6k shadow-probe log lines — per note-open). This locks the +//! two-stage shape: **one** O(notes) centroid scan, then one bounded per-note vector +//! fetch per shortlisted note. //! //! Sole test in its binary on purpose: it installs a scoped tracing subscriber to read //! SQLite's per-statement profiler back, and tracing's global callsite-interest cache @@ -44,14 +44,21 @@ impl<'a> MakeWriter<'a> for Capture { /// The exact template `db::note_for_chunk` emits — a per-hit chunk→note resolution. const PER_HIT_SQL: &str = "SELECT note_b2id FROM chunks WHERE id = ?1"; -/// The whole-space scan `db::for_each_stored_vector` emits — must run **once** for the -/// whole call, not once per anchor chunk (the old per-anchor KNN scan storm). -const SPACE_SCAN_SQL: &str = "SELECT chunk_id, embedding FROM chunks_vec"; +/// The stage-1 coarse scan `db::for_each_note_centroid` emits — must run **once** +/// per call: it is the only whole-space read discovery is allowed. +const CENTROID_SCAN_SQL: &str = "SELECT note_b2id, centroid FROM note_centroids"; +/// The whole-space chunk-vector scan (`db::for_each_stored_vector`) — search's +/// primitive, which discovery must **never** run: reading every stored vector per +/// note-open is exactly the O(vault) cost #38 removed. +const SPACE_SCAN_SQL: &str = "SELECT chunk_id, vector FROM embeddings"; +/// The per-note vector fetch (`db::note_chunk_vectors`) — stage 2's unit, allowed +/// once for the anchor plus once per *shortlisted note*, never per chunk. +const PER_NOTE_SQL: &str = "SELECT c.id, e.vector FROM chunks c JOIN embeddings e"; #[test] -fn candidates_resolves_notes_without_a_per_hit_query() { - // A multi-chunk anchor plus several multi-chunk notes: a per-hit resolution would - // fire anchor_chunks × total_chunks times (a scaled-down mirror of the real vault). +fn candidates_issues_bounded_sql_never_o_chunks() { + // Multi-chunk notes so any per-chunk statement pattern would visibly exceed the + // note count (a scaled-down mirror of the real vault). const NOTES: usize = 12; const PARAS: usize = 5; // blank-line-separated paragraphs → ~1 chunk each @@ -86,13 +93,13 @@ fn candidates_resolves_notes_without_a_per_hit_query() { ) .unwrap(); assert!( - anchor_chunks > 1 && total_chunks > anchor_chunks, - "the N+1 only shows with a multi-chunk anchor over a larger vault \ + anchor_chunks > 1 && total_chunks > NOTES as i64, + "O(chunks) patterns only show with multi-chunk notes \ (anchor={anchor_chunks}, total={total_chunks})" ); // Run candidates under a DEBUG JSON subscriber so SQLite's per-statement profiler - // (target b2::sqlite) is captured; count the per-hit resolution template. + // (target b2::sqlite) is captured; count the statement templates. let capture = Capture::default(); let subscriber = tracing_subscriber::fmt() .json() @@ -108,22 +115,34 @@ fn candidates_resolves_notes_without_a_per_hit_query() { let text = String::from_utf8(capture.0.lock().unwrap().clone()).unwrap(); - // (1) chunk→note resolution is one bulk load, not a per-hit query. + // (1) chunk→note resolution never happens per hit (stage 2 works note-by-note, + // so it needs no resolution at all). let per_hit = text.matches(PER_HIT_SQL).count(); - let n_plus_1 = (anchor_chunks * total_chunks) as usize; - assert!( - per_hit <= 1, - "chunk→note resolution must be one bulk query, not per hit: saw {per_hit} \ - `note_for_chunk` statements (an N+1 would be anchor×vault = \ - {anchor_chunks}×{total_chunks} = {n_plus_1})" + assert_eq!( + per_hit, 0, + "discovery must not resolve chunk→note per hit (saw {per_hit})" + ); + + // (2) exactly one whole-space read, and it is the O(notes) centroid scan. + let centroid_scans = text.matches(CENTROID_SCAN_SQL).count(); + assert_eq!( + centroid_scans, 1, + "the coarse centroid scan must run exactly once per call" ); - // (2) the vector space is scanned exactly once, not once per anchor chunk — the - // old per-anchor KNN scan reread the whole ~100k-vector space `anchor_chunks` times. + // (3) the O(chunks) whole-space vector scan never runs on the note-open path. let space_scans = text.matches(SPACE_SCAN_SQL).count(); assert_eq!( - space_scans, 1, - "the whole-space scan must run once for the call, not once per anchor chunk \ - (anchor_chunks = {anchor_chunks})" + space_scans, 0, + "discovery must not scan every stored chunk vector (#38): saw {space_scans}" + ); + + // (4) per-note vector fetches are bounded by the shortlist (≤ one per note in + // this small vault: the anchor + every candidate), never by the chunk count. + let per_note = text.matches(PER_NOTE_SQL).count(); + assert!( + (1..=NOTES).contains(&per_note), + "stage-2 fetches must be one per shortlisted note (≤ {NOTES}), got {per_note} \ + (an O(chunks) pattern would approach {total_chunks})" ); } diff --git a/crates/b2-core/tests/embed.rs b/crates/b2-core/tests/embed.rs index 3803272..45b2da5 100644 --- a/crates/b2-core/tests/embed.rs +++ b/crates/b2-core/tests/embed.rs @@ -1,7 +1,8 @@ -//! Step 3 — `sqlite-vec` + the embedder seam +//! Step 3 — the vector store + the embedder seam //! (planning/specs/completed/index-engine-build.md step 3): a deterministic fake embedder //! produces reproducible KNN; `embed_model_id`/`embed_dim` are recorded; a -//! model/dim swap recreates the vector space. +//! model/dim swap recreates the vector space; note centroids (discovery's coarse +//! stage, #38) track the stored chunk vectors. mod common; @@ -90,7 +91,7 @@ fn reindex_with_progress_reports_cumulative_and_fully_embeds() { // Batched embed still populates a vector for every chunk. let total = count(&conn, "chunks"); assert!(total > 0); - assert_eq!(count(&conn, "chunks_vec"), total); + assert_eq!(count(&conn, "embeddings"), total); // Progress: reported, per-note fields populated, notes_embedded within the // stable denominator and monotonic, chunks_done non-decreasing and ending @@ -148,13 +149,13 @@ fn reindex_is_incremental_and_force_reembeds_everything() { } #[test] -fn ingest_populates_chunks_vec_and_records_meta() { +fn ingest_populates_embeddings_and_records_meta() { let tmp = tempfile::TempDir::new().unwrap(); let conn = ingest_golden(tmp.path(), &FakeEmbedder::new(64)); // one vector per chunk assert!(count(&conn, "chunks") > 0); - assert_eq!(count(&conn, "chunks"), count(&conn, "chunks_vec")); + assert_eq!(count(&conn, "chunks"), count(&conn, "embeddings")); assert_eq!( meta(&conn, "embed_model_id").as_deref(), @@ -163,6 +164,68 @@ fn ingest_populates_chunks_vec_and_records_meta() { assert_eq!(meta(&conn, "embed_dim").as_deref(), Some("64")); } +/// `note_centroids` is derived data with the vectors' own lifecycle: after any embed +/// pass, every note with stored vectors carries a centroid, and it equals +/// `centroid_of` over exactly those vectors — including after a body edit re-chunks +/// and re-embeds the note (the stale centroid must not survive). +#[test] +fn centroids_track_the_stored_chunk_vectors() { + use b2_core::embed::{centroid_of, pack_f32}; + use b2_core::vault::Vault; + + let tmp = tempfile::TempDir::new().unwrap(); + let root = tmp.path().join("vault"); + golden_vault_copy(&root); + let vault = Vault::open(&root).unwrap(); + vault.reindex().unwrap(); + + let conn = open(&root.join(".b2").join("b2.sqlite")).unwrap(); + let assert_centroids_current = |conn: &Connection| { + let notes_with_vectors: i64 = conn + .query_row( + "SELECT COUNT(DISTINCT c.note_b2id) FROM chunks c + JOIN embeddings e ON e.chunk_id = c.id", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!( + count(conn, "note_centroids"), + notes_with_vectors, + "one centroid per embedded note" + ); + let mut stmt = conn + .prepare("SELECT note_b2id, centroid FROM note_centroids") + .unwrap(); + let rows: Vec<(String, Vec)> = stmt + .query_map([], |r| Ok((r.get(0)?, r.get(1)?))) + .unwrap() + .collect::>() + .unwrap(); + for (note, stored) in rows { + let vectors: Vec> = db::note_chunk_vectors(conn, ¬e) + .unwrap() + .into_iter() + .map(|(_, v)| v) + .collect(); + let expected = centroid_of(&vectors).expect("an embedded note has vectors"); + assert_eq!( + stored, + pack_f32(&expected), + "centroid of {note} summarizes its current vectors" + ); + } + }; + assert_centroids_current(&conn); + + // Edit one note's body → re-project + re-embed → its centroid must follow. + let srs = root.join("notes/spaced-repetition.md"); + let text = std::fs::read_to_string(&srs).unwrap(); + std::fs::write(&srs, format!("{text}\n\nFreshly appended centroid bait.")).unwrap(); + vault.reindex().unwrap(); + assert_centroids_current(&conn); +} + #[test] fn knn_finds_the_chunk_whose_text_we_query() { let tmp = tempfile::TempDir::new().unwrap(); @@ -198,7 +261,7 @@ fn reindex_yields_identical_vectors() { let vec_for_srs_seq0 = |c: &Connection| -> Vec { c.query_row( - "SELECT v.embedding FROM chunks_vec v + "SELECT v.vector FROM embeddings v JOIN chunks c ON c.id = v.chunk_id WHERE c.note_b2id = ?1 AND c.seq = 0", [SRS_ID], @@ -219,15 +282,21 @@ fn reindex_yields_identical_vectors() { fn changing_dim_recreates_the_vector_space_and_clears_vectors() { let tmp = tempfile::TempDir::new().unwrap(); let conn = ingest_golden(tmp.path(), &FakeEmbedder::new(64)); - assert!(count(&conn, "chunks_vec") > 0); + assert!(count(&conn, "embeddings") > 0); // A model/dim swap: the only place it can be detected is meta. Vectors are - // dropped (a full re-embed is required) and the dim is updated. + // dropped (a full re-embed is required) and the dim is updated. Centroids share + // the vectors' lifecycle, so the swap empties them too. db::ensure_embedding_space(&conn, "fake-deterministic-v1", 128).unwrap(); assert_eq!(meta(&conn, "embed_dim").as_deref(), Some("128")); assert_eq!( - count(&conn, "chunks_vec"), + count(&conn, "embeddings"), 0, "swap drops vectors; re-embed needed" ); + assert_eq!( + count(&conn, "note_centroids"), + 0, + "swap drops centroids with the vectors they summarize" + ); } diff --git a/crates/b2-core/tests/project_embed.rs b/crates/b2-core/tests/project_embed.rs index 98c6a2c..54cc044 100644 --- a/crates/b2-core/tests/project_embed.rs +++ b/crates/b2-core/tests/project_embed.rs @@ -32,7 +32,7 @@ fn project_only_builds_keyword_graph_index_with_no_vectors() { let conn = open(&tmp.path().join("b2.sqlite")).unwrap(); // Projection alone: no embedder anywhere near the call. If it issued any query - // against `chunks_vec` (which does not exist yet), this would error. + // against `embeddings` (which does not exist yet), this would error. let outcome = project_vault(&conn, &vault_dir, &UlidGen, false).unwrap(); assert_eq!(outcome.notes.len(), 2); @@ -48,7 +48,7 @@ fn project_only_builds_keyword_graph_index_with_no_vectors() { // …and the embedding space was never created (that is the embed pass's job). assert!( !db::embedding_space_exists(&conn).unwrap(), - "projection must not create chunks_vec" + "projection must not create the vector tables" ); } @@ -66,13 +66,13 @@ fn embed_fills_exactly_the_missing_vectors() { let first = embed_vault(&conn, &embedder, &mut |_| ControlFlow::Continue(())).unwrap(); assert!(!first.cancelled); assert_eq!(first.embedded.len(), 2, "both projected notes embed"); - assert_eq!(count(&conn, "chunks_vec"), count(&conn, "chunks")); + assert_eq!(count(&conn, "embeddings"), count(&conn, "chunks")); // Second embed: the DB-derived pending set is empty → fills 0, changes nothing. let second = embed_vault(&conn, &embedder, &mut |_| ControlFlow::Continue(())).unwrap(); assert!(!second.cancelled); assert!(second.embedded.is_empty(), "a second embed fills nothing"); - assert_eq!(count(&conn, "chunks_vec"), count(&conn, "chunks")); + assert_eq!(count(&conn, "embeddings"), count(&conn, "chunks")); } /// The observable projection of an index: note count, `(note, seq) → chunk text`, @@ -105,8 +105,8 @@ fn observable_state(root: &Path) -> Observable { let text_to_vector = { let mut stmt = conn .prepare( - "SELECT c.text, v.embedding FROM chunks c - JOIN chunks_vec v ON v.chunk_id = c.id + "SELECT c.text, v.vector FROM chunks c + JOIN embeddings v ON v.chunk_id = c.id ORDER BY c.note_b2id, c.seq", ) .unwrap(); diff --git a/crates/b2-core/tests/substrate.rs b/crates/b2-core/tests/substrate.rs index c46bc81..9c43043 100644 --- a/crates/b2-core/tests/substrate.rs +++ b/crates/b2-core/tests/substrate.rs @@ -2,19 +2,18 @@ //! //! Green-scenario assertions for build-plan step 0 //! (planning/specs/completed/index-engine-build.md §4): -//! - `sqlite-vec` statically links; FTS5 is compiled in (the `bundled` SQLite). -//! - open→reopen is stable; `WAL` + `foreign_keys=ON` hold; `schema_version` seeded. -//! -//! This is the riskiest integration in the whole design (index-engine.md §3): if -//! FTS5 (BM25) and `sqlite-vec` (KNN) can't live in one connection, the -//! single-store premise is wrong. So it is the first thing we prove. +//! - FTS5 is compiled in (the `bundled` SQLite). *(Vectors need no substrate proof +//! since schema v3, #38: they are plain BLOB tables scored in-process — the +//! `sqlite-vec` half of the original bet was retired with the dependency.)* +//! - open→reopen is stable; `WAL` + `foreign_keys=ON` hold; the #38 scan pragmas +//! (`mmap_size`/`cache_size`) are applied; `schema_version` seeded. use b2_core::{open, SCHEMA_VERSION}; -/// The load-bearing bet: BM25 full-text search AND brute-force vector KNN, in the -/// *same* statically-linked connection, no runtime `load_extension`. +/// The load-bearing bet: BM25 full-text search in the statically-linked bundled +/// SQLite, no runtime `load_extension`. #[test] -fn fts5_and_sqlite_vec_coexist_in_one_connection() { +fn fts5_works_in_the_bundled_connection() { let tmp = tempfile::TempDir::new().unwrap(); let conn = open(&tmp.path().join("b2.sqlite")).unwrap(); @@ -33,22 +32,6 @@ fn fts5_and_sqlite_vec_coexist_in_one_connection() { ) .unwrap(); assert_eq!(hit, 1, "BM25 should rank the memory note first"); - - // sqlite-vec present, brute-force KNN over a vec0 virtual table works. - conn.execute_batch( - "CREATE VIRTUAL TABLE v USING vec0(embedding FLOAT[3]); - INSERT INTO v(rowid, embedding) VALUES (1, '[0.10, 0.20, 0.30]'); - INSERT INTO v(rowid, embedding) VALUES (2, '[0.90, 0.80, 0.70]');", - ) - .unwrap(); - let nearest: i64 = conn - .query_row( - "SELECT rowid FROM v WHERE embedding MATCH '[0.11, 0.19, 0.31]' ORDER BY distance LIMIT 1", - [], - |r| r.get(0), - ) - .unwrap(); - assert_eq!(nearest, 1, "KNN should return the nearest vector"); } /// The locked pragmas and the `meta` bookkeeping survive a close/reopen, and the @@ -68,6 +51,16 @@ fn pragmas_and_schema_version_persist_across_reopen() { .query_row("PRAGMA foreign_keys", [], |r| r.get(0)) .unwrap(); assert_eq!(foreign_keys, 1, "foreign_keys must be ON"); + // The #38 read-path pragmas: whole-space vector scans must stream through + // the OS page cache (mmap), not a pread-per-page under the 2 MB default. + let mmap_size: i64 = conn + .query_row("PRAGMA mmap_size", [], |r| r.get(0)) + .unwrap(); + assert!(mmap_size > 0, "mmap_size must be engaged, got {mmap_size}"); + let cache_size: i64 = conn + .query_row("PRAGMA cache_size", [], |r| r.get(0)) + .unwrap(); + assert_eq!(cache_size, -32768, "cache_size must be raised (KiB units)"); } // connection dropped → file closed // Reopen: schema_version is stable and not duplicated. diff --git a/crates/b2-core/tests/vector_pool_scale.rs b/crates/b2-core/tests/vector_pool_scale.rs index af4d8cf..3134103 100644 --- a/crates/b2-core/tests/vector_pool_scale.rs +++ b/crates/b2-core/tests/vector_pool_scale.rs @@ -1,13 +1,13 @@ -//! Regression: the full-space vector callers must not trip `sqlite-vec`'s KNN -//! `k` ceiling. `discover::candidates` and `search::graph_filtered_search` scan -//! the *whole* embedding space (index-engine.md §4 promises brute-force is -//! comfortable at 50–100k chunks). The `vec0` `MATCH … LIMIT k` operator refuses -//! `k > 4096` ("k value in knn query too large"), so past ~4096 chunks those two -//! commands — and a `search` with a large `--limit` — crashed. They must instead -//! degrade to a full brute-force scan and return results. +//! Scale smoke: discovery, graph-filtered search, and an oversized `--limit` must +//! all survive a vault of thousands of chunks and return sane, un-truncated +//! results. Historically this locked the fix for `sqlite-vec`'s `vec0` +//! `MATCH … LIMIT k` ceiling (`k > 4096` → "k value in knn query too large", which +//! crashed all three); the vec0 store is gone (schema v3, #38 — plain tables, +//! in-process scoring, two-stage discovery), but the boundary stays exercised so no +//! silent cap ever returns to these paths. //! //! Scope: with the deterministic *fake* embedder this proves the plumbing (no -//! crash, sane results) across the 4096-chunk boundary, not model quality. +//! crash, sane results) well past that historical boundary, not model quality. mod common; @@ -19,7 +19,8 @@ use rusqlite::Connection; use std::fs; use std::path::Path; -/// `sqlite-vec`'s hard KNN cap; we build a vault comfortably past it. +/// The retired vec0 store's hard KNN cap; we keep building vaults comfortably past +/// it so these whole-space paths stay proven at thousands of chunks. const KNN_CAP: usize = 4096; /// Build a vault of `notes` unlinked notes, each with `paras` blank-line-separated diff --git a/crates/b2-core/tests/write.rs b/crates/b2-core/tests/write.rs index 40b87b0..625a657 100644 --- a/crates/b2-core/tests/write.rs +++ b/crates/b2-core/tests/write.rs @@ -116,7 +116,7 @@ fn write_reprojects_keyword_graph_and_clears_stale_vectors() { let tmp = tempfile::TempDir::new().unwrap(); let (vault, root) = reindexed(tmp.path()); let conn = index_conn(&root); - assert_eq!(count(&conn, "chunks_vec"), count(&conn, "chunks")); + assert_eq!(count(&conn, "embeddings"), count(&conn, "chunks")); // The golden SRS note links memory twice (references + elaborates). Save a body // that keeps ONE link and adds fresh text. @@ -145,12 +145,12 @@ fn write_reprojects_keyword_graph_and_clears_stale_vectors() { assert!(missing.iter().all(|(_, path, _, _)| path == SRS_PATH)); let embed = vault.embed(&mut |_| ControlFlow::Continue(())).unwrap(); assert_eq!(embed.embedded, 1, "the embed pass fills the saved note"); - assert_eq!(count(&conn, "chunks_vec"), count(&conn, "chunks")); + assert_eq!(count(&conn, "embeddings"), count(&conn, "chunks")); } #[test] fn write_needs_no_embedding_space() { - // A projected-only vault (no chunks_vec, no model anywhere): saving works — the + // A projected-only vault (no vector tables, no model anywhere): saving works — the // model-free proof (§7 invariant 4). let tmp = tempfile::TempDir::new().unwrap(); let root = tmp.path().join("vault"); diff --git a/crates/b2-embed/src/model.rs b/crates/b2-embed/src/model.rs index b234d7b..83da441 100644 --- a/crates/b2-embed/src/model.rs +++ b/crates/b2-embed/src/model.rs @@ -99,7 +99,7 @@ impl LocalEmbedder { } /// The pooled, L2-normalized embedding of `text`. CLS pooling (row 0) — what - /// bge is trained for; normalized so `sqlite-vec`'s L2 distance ranks by cosine. + /// bge is trained for; normalized so the index's L2 distance ranks by cosine. fn embed_inner(&self, text: &str) -> candle_core::Result> { let enc = self .tokenizer diff --git a/planning/index-engine.md b/planning/index-engine.md index 98a6154..de2eef1 100644 --- a/planning/index-engine.md +++ b/planning/index-engine.md @@ -214,6 +214,16 @@ Reality check on `sqlite-vec` so we don't over-promise: So: **semantic search ships in v1**, brute-force KNN, 768-dim float vectors, with quantization in our back pocket. This is the headline consequence of choosing SQLite. +**Update (2026-07-12, [#38](https://github.com/AlteredCraft/B2/issues/38)).** The brute-force *math* +scaled as predicted; `sqlite-vec`'s **read path** did not — every `vec0` scan probes a shadow table +per row (~38.6k internal statements per `b2 similar` on the primary vault, ~4.4 s per note-open). +Since its only shipped search was brute force we already compute, the dependency was **removed**: +vectors now live in plain tables (`embeddings`, plus per-note `note_centroids`) scored in-process, +and discovery is two-stage (O(notes) centroid shortlist → exact max-sim rescore). Same SQLite store, +same single-store property, same exact results at test scale. Full analysis + options: +[research/discovery-scan-strategy.md](research/discovery-scan-strategy.md). Quantization and ANN keep +their standby order, now behind the centroid stage. + ## 5. The reranker as a fast follow Slot it exactly where qmd puts it: **after RRF fusion, before final ranking**, behind a swappable seam. diff --git a/planning/research/discovery-scan-strategy.md b/planning/research/discovery-scan-strategy.md new file mode 100644 index 0000000..953f8fb --- /dev/null +++ b/planning/research/discovery-scan-strategy.md @@ -0,0 +1,200 @@ +--- +title: "B2 — Research: making discovery O(fast) — the #38 scan-strategy decision" +type: note +tags: [b2, research, index-engine, sqlite-vec, discovery, performance, quantization, ann] +created: 2026-07-12 +status: draft +--- + +# B2 — Research: making discovery O(fast) — the #38 scan-strategy decision + +> Deep analysis for [#38](https://github.com/AlteredCraft/B2/issues/38): note-open discovery +> (`b2 similar`, the desktop **Similar & Connections** pane) is still ~4.4 s (release) on the +> primary vault after the [#37](https://github.com/AlteredCraft/B2/issues/37) fix. Goal: relation +> resolution for a given document feels **instant**, and stays instant at **10× the current +> document population** (~1k docs → ~10k docs). Constraints: local-first on a laptop, no +> app-managed result cache (OS/library-internal caches are fine), background one-time heavy +> lifts acceptable, maintained derived structures acceptable if low-effort. Full refactor was +> on the table; the conclusion is it isn't needed. + +## TL;DR / recommendation + +**Own the flat scan: store vectors in a plain SQLite table, score them in-process, and drop +`vec0` from the read path entirely. Then make discovery two-stage (note-centroid prefilter → +exact chunk rescore) so the heavy pass is O(notes), not O(chunks).** No new store, no ANN +library, no managed cache. + +1. **Now (fixes #38 and #36 structurally):** replace `chunks_vec` (a `vec0` virtual table) with + a plain `embeddings(chunk_id INTEGER PRIMARY KEY, vector BLOB)` table; scan it with one + sequential statement; fix the arithmetic shape (reused scratch buffer, unrolled accumulators); + add `PRAGMA mmap_size` + a bigger `cache_size` in `db::open`. Expected: **~4.4 s → ~0.2–0.3 s + warm** at today's scale, bit-identical ranking, one SQL log line per open instead of ~38.6K, + and the `sqlite-vec` dependency can be removed outright. +2. **For 10× (small; can ship with 1):** a per-note **centroid** column maintained by the same + embed pass — discovery scans ~N_notes centroids (10k × 768 ≈ 30 MB at 10×, ~5 ms), shortlists + the top ~200 notes, and runs today's exact max-sim only over the shortlist's chunks + (~10–20 ms). Scaling becomes essentially flat in vault size. +3. **Multiplier already on the roadmap:** the qmd chunker upgrade + ([#19](https://github.com/AlteredCraft/B2/issues/19)) shrinks ~38 paragraph-chunks/doc to a + handful of ~900-token chunks — a ~5–40× reduction in vector count that makes every option + cheaper (and max-sim less noisy). Do it for quality; bank the perf. +4. **Held in reserve, in order:** int8/binary **quantization** (shrinks the constant, needs the + eval), **ANN** (usearch/LanceDB — wrong scale, non-deterministic, standing maintenance; + both planning docs already ranked it last), **precomputed similar-notes table** (a managed + cache with an O(N²)-shaped invalidation problem — the explicit last resort). + +## 1. Problem decomposition — which "relations" are slow + +"Resolve the relations of a given document" is two different queries: + +- **Authored relations** (backlinks + outbound, `Vault::neighbors`/`explain`) — indexed lookups + on the materialized `edges` table. Already O(link-degree), microseconds. **Not the problem.** +- **Latent relations** (`b2 similar` / `discover::candidates` — "semantically near ∖ already + connected") — an **exact brute-force max-sim scan of every stored chunk vector on every + note-open**. This is the whole problem, and (post-#37) it is the only remaining O(vault) + read in the open-a-note path. `hybrid_search` and `graph_filtered_search` pay the same + per-row tax on their `vec0` scans, so whatever fixes discovery fixes them for free. + +## 2. What we've learned so far (the #35→#36→#37 trail) + +The issue history localizes the cost precisely; this is worth restating because it *rules out* +most exotic fixes: + +- **#37:** the A× whole-space rescan (one KNN scan per anchor chunk) and the per-hit N+1 were + the first-order bug — fixing the *access pattern* took 51 s → 4.4 s with zero algorithm change. +- **#36 + #38:** what remains is dominated by **how the bytes are read, not how many FLOPs are + done**. Every row read from `chunks_vec` makes `sqlite-vec` probe its `chunks_vec_rowids` + shadow table (~38.6K internal single-row statements per open — the O(vault) log lines), and + the 132 MB DB is walked through SQLite's default ~2 MB page cache with no mmap (the ~3.2 s + that stays *system* time even warm). +- **Arithmetic shape, measured** (microbench at N=38,610 × 768-dim, A=12, this container's CPU, + `rustc -O`): the current shape — per-row `unpack_f32` `Vec` allocation + iterator-`sum()` + `l2_sq` (float non-associativity blocks autovectorization) — costs **~530 ms**; a reused + scratch buffer + 8-accumulator unrolled loop costs **~75 ms** (10×: ~5.2 s vs ~0.74 s). The + index-engine.md §4 prediction ("tens of ms") was about the FLOPs and remains right; the + implementation shape multiplies it ~7×. +- **Structural fact:** `sqlite-vec`'s shipped search is brute force — no ANN path — and + discovery already scores in-process. So on the scan path `vec0` provides **storage only**, + at the price of a per-row vtab round-trip. B2 uses none of what it would charge for. +- **Structural fact #2:** the shipped chunker is the *minimal paragraph splitter* (`chunk.rs`), + not the planned ~900-token qmd heuristic (#19). ~1k docs → 38.6k chunks (~38/doc) means the + vector population is inflated ~5–40× over the design target. #38's cost is partly a chunking + artifact. + +Consequence: **before changing the algorithm, the read path and arithmetic shape are worth +~10–20×.** That reframes the option space — heuristics are for the 10× headroom, not for today's +4.4 s. + +## 3. Budget at target scale + +| | today (~1k docs) | 10× (~10k docs, current chunker) | 10× + #19 chunker | +|---|---|---|---| +| chunks | ~38.6k | ~386k | ~30–80k | +| f32 vectors on disk | ~118 MB | ~1.2 GB | ~90–240 MB | +| flat scan, fixed shape (measured/extrapolated) | ~75 ms + read | ~740 ms + read | ~60–160 ms + read | +| two-stage (centroid → rescore ~200 notes) | ~10 ms | **~15–25 ms** | ~10 ms | + +Brute force with a sane memory layout is comfortably "instant" today and *borderline* at 10× +under the current chunker; the centroid stage (or #19 alone) restores a wide margin. This +matches industry practice: FAISS's own guidance is flat/exact search until high-hundreds-of- +thousands of vectors, and ANN only when exactness or memory must be traded for scale. A +personal vault never leaves the flat regime — *if* the flat scan isn't paying a per-row +virtual-table tax. + +## 4. Options considered + +### A. Own the flat scan (plain table + in-process top-k + pragmas) — **do now** + +Store embeddings in an ordinary table (`embeddings(chunk_id INTEGER PRIMARY KEY, vector BLOB)`, +same packed-LE-f32 blob), keyed/lifecycled exactly as `chunks_vec` is today (created at embed +time at the model's dim; dropped on model swap; `meta` discipline unchanged). All four vector +consumers — `for_each_stored_vector`, `vector_search`, `vector_search_all`, and #36's +missing-vector anti-join — become one sequential B-tree scan (plus, for search, an in-process +top-k heap). Add `PRAGMA mmap_size` (≥ DB size, e.g. 1–2 GB — it's a cap, not an allocation) +and a bigger `page_size`/`cache_size` so the scan streams from the OS page cache — the one +cache we're happy to use because nobody manages it. Fix `l2_sq`/`unpack_f32` shape as measured. + +- **Pros:** simplest possible change; results bit-identical (same vectors, same metric, same + order); kills the O(vault) log lines and #36's probe storm *structurally*; deletes a pre-v1 + dependency (`sqlite-vec`) instead of adding one; keeps the single-store `chunks ⨝ edges ⨝ + embeddings` join property that decided the qmd and Chroma/Lance evaluations; fully + deterministic; no index to maintain beyond what embed already maintains. +- **Cons:** we own ~50 lines of scan/top-k code and the (already-shared) blob format; still + O(chunks) per open — at 10× under the current chunker that's ~0.8–1 s warm, i.e. A alone + doesn't hit the 10× "instant" bar; forfeits a hypothetical future sqlite-vec ANN mode (which + the planning docs already said not to design for). + +### B. Two-stage discovery: note-centroid prefilter → exact chunk rescore — **the 10× lever** + +At embed time, also store one **summary vector per note** (mean of its chunk vectors, +L2-renormalized; summation in `seq` order so it's deterministic). Discovery becomes: (1) scan +~N_notes centroids against the anchor's centroid, excluding the anchor + 1-hop neighbors; +(2) take a generous shortlist (~10–20× the display limit, e.g. 200–300 notes); (3) run today's +exact max-sim over only the shortlist's chunks; surface evidence chunks as now. This is IVF's +coarse-quantizer idea using B2's *natural* partition (the note) instead of learned clusters — +no training, no ANN library, no recall cliff. + +- **Pros:** the heavy pass drops from O(chunks) to O(notes) — ~30 MB / ~5 ms at 10×, flat + effectively forever (100k notes ≈ 300 MB ≈ still sub-100 ms); maintenance is one UPDATE per + note inside the existing embed pass (the same lifecycle as the vectors themselves — a derived + projection in the disposable index, **not** an app-managed result cache: nothing to + invalidate that embed doesn't already touch); deterministic; the note-level vector is also a + natural substrate for #20 (distance weighting) later. +- **Cons:** no longer exhaustively exact — a note whose centroid is far away but which contains + one very close paragraph can miss the shortlist. Mitigations: discovery is *by design* + permissive/recall-oriented with a human precision gate, the shortlist is cheap to widen, and + the retrieval eval (eval-strategy.md) can pin shortlist size against exact-scan ground truth. + Slightly more code than A (two stages instead of one). + +### C. Quantization (int8 / binary + f32 rescore) — **reserve** + +`index-engine.md` §4 names it the first lever before ANN. Once we own the scan (A), an extra +int8 (4×) or binary (32×, Hamming/popcount, rescore top-k in f32) column is easy to add and is +standard practice for bge-family embeddings. + +- **Pros:** shrinks bytes-read and FLOPs by 4–32× (1.2 GB → 300 MB / 37 MB at 10×); composes + with A and B; well-trodden (FAISS SQ8, binary-quantized bge retains ~95 %+ recall with + rescoring). +- **Cons:** measurable quality risk → gated on the eval harness; more code than B for a smaller + win than B at this scale; reduces the constant, not the O(chunks) growth. Only worth it if, + after A+B(+#19), cold-start I/O or memory footprint still bites. + +### D. Real ANN (HNSW via usearch/hnsw_rs, IVF, or LanceDB vector tier) — **rejected** + +- **Pros:** sub-linear queries; the industry default at 10M+ vectors. +- **Cons:** solves a scale B2 doesn't reach at 10× (§3); HNSW graphs are build-order-dependent + (breaks the determinism requirement); a second index artifact to build/persist/heal alongside + the disposable-projection invariant; dependency weight vs. the single binary; both + `index-engine.md` §4 and `research/vector-store-alternatives.md` already ranked ANN last and + re-affirmed staying on SQLite. Nothing in #38's evidence (read-path overhead, not FLOPs) + argues for it. + +### E. Materialized similar-notes table (precompute discovery per note) — **last resort, as specified** + +Background-compute top-k candidates per note at embed time; note-open reads a row. + +- **Pros:** O(1) note-open, trivially instant. +- **Cons:** this is exactly the app-managed cache the requirements deprioritize: one note's new + embedding can perturb *every* other note's candidate list, so correctness needs either O(N²) + recompute sweeps or push-based incremental maintenance — a standing effort/complexity bill. + B delivers ~tens-of-ms opens without any of that; E only becomes interesting if the target + were single-digit ms on a vastly larger corpus. + +## 5. Sequencing & migration + +1. **A** — new `embeddings` table + in-process top-k + pragmas + arithmetic shape; delete + `vec0`/`sqlite-vec`. Bump `schema_version`: the existing migration gate drops derived tables + and the next `reindex` rebuilds — and thanks to the projection/embedding split the app stays + usable (BM25 + graph) while the background embed refills vectors. That *is* the acceptable + "one-time heavy lift, slightly diminished state". (If re-embedding ~38.6k chunks is + undesirable, a one-shot copy-across from `chunks_vec` before dropping it avoids even that.) + Closes #38's acceptance criteria at today's scale and #36 as a side effect. +2. **B** — centroid column + two-stage `discover::candidates`, shortlist size pinned by the + eval against exact ground truth. Buys the 10× (and 100×) headroom. +3. **#19** — the chunker upgrade, on its own quality-driven schedule; it multiplies both wins. +4. **C** then **D** stay shelved until the eval + measured latency say otherwise; **E** stays + the last resort. + +Regression guardrails, matching the #37 pattern: extend `discover_query_count.rs` to assert the +whole-space pass emits O(1) SQL statements (not O(chunks)), and add a scale smoke-test with a +synthetic few-hundred-note vault under the fake embedder. diff --git a/ui/src/main.ts b/ui/src/main.ts index 878c2d1..4ed7612 100644 --- a/ui/src/main.ts +++ b/ui/src/main.ts @@ -148,14 +148,17 @@ async function openNote(ref: string): Promise { state.similar = []; state.connections = []; state.loading = false; - state.discovering = true; + state.discoveringSimilar = true; + state.discoveringConnections = true; render(); await refreshDiscovery(); } catch (e) { flash(errText(e)); } finally { + // The discovery flags are owned by refreshDiscovery (it clears each section's when + // that read settles, guarded against a superseding open) — clearing them here would + // race a newer note's in-flight load, so only the middle-pane spinner is ours. state.loading = false; - state.discovering = false; render(); } } @@ -189,24 +192,45 @@ function toggleSource(): void { async function refreshDiscovery(): Promise { const n = state.current; if (!n) return; - try { - const [similar, explain] = await Promise.all([ - api.similar(n.path), - api.explain(n.path), - ]); - // Now that discovery is awaited off the render path, the user can navigate away - // while it's in flight — don't clobber the new note's side pane with a stale result. - if (state.current?.path !== n.path) return; - state.similar = similar; - state.connections = explain.connections; - } catch (e) { - // Discovery failing (e.g. an unembedded vault) is non-fatal — show the note, - // empty the panes, and surface the reason. - if (state.current?.path !== n.path) return; - state.similar = []; - state.connections = []; - flash(errText(e)); - } + // Two independent reads with independent repaints: `explain` (Connections) is a + // near-instant graph read, `similar` is the slower whole-vault discovery scan. A + // Promise.all would gate the fast one on the slow one — so each settles and paints on + // its own. Both guard against the user having navigated away before they resolved + // (don't clobber the new note's pane) and clear only their own section's loading flag. + const stale = () => state.current?.path !== n.path; + const connections = api + .explain(n.path) + .then((explain) => { + if (!stale()) state.connections = explain.connections; + }) + .catch((e) => { + if (!stale()) { + state.connections = []; + flash(errText(e)); + } + }) + .finally(() => { + if (stale()) return; + state.discoveringConnections = false; + render(); + }); + const similar = api + .similar(n.path) + .then((cands) => { + if (!stale()) state.similar = cands; + }) + .catch((e) => { + if (!stale()) { + state.similar = []; + flash(errText(e)); + } + }) + .finally(() => { + if (stale()) return; + state.discoveringSimilar = false; + render(); + }); + await Promise.all([connections, similar]); } async function doSearch(raw: string): Promise { diff --git a/ui/src/render.ts b/ui/src/render.ts index ef9bc5a..d68c2d1 100644 --- a/ui/src/render.ts +++ b/ui/src/render.ts @@ -264,8 +264,11 @@ function discoverySectionHtml(state: AppState): string { function similarSectionHtml(state: AppState): string { const head = `

Similar & unlinked

`; if (state.similar.length === 0) { - if (state.discovering) - return head + `

Finding similar notes…

`; + if (state.discoveringSimilar) + return ( + head + + `
` + ); const hint = state.semantic ? "Nothing similar-but-unlinked, or the vault isn’t embedded yet (Reindex)." : "Semantic similarity is off — run b2 init then Reindex."; @@ -294,7 +297,7 @@ function connectionsSectionHtml(state: AppState): string { return ( head + `

${ - state.discovering ? "Loading connections…" : "No connections yet." + state.discoveringConnections ? "Loading connections…" : "No connections yet." }

` ); const items = state.connections diff --git a/ui/src/state.ts b/ui/src/state.ts index 3500a24..6c6727d 100644 --- a/ui/src/state.ts +++ b/ui/src/state.ts @@ -65,12 +65,14 @@ export interface AppState { /** The open note's typed edges (from explain). */ connections: NeighborView[]; /** - * Discovery (`similar` + `explain`) is in flight for the open note. Kept separate - * from `loading` so the note body paints the instant it's read — discovery is a - * slower, independent side-pane read and must not gate the middle pane. Drives the - * side pane's "finding…" hint so an empty pane mid-load doesn't read as "nothing found". + * Discovery reads in flight for the open note, tracked **per side-pane section** so + * the fast graph read (`explain` → Connections) paints without waiting on the slower + * whole-vault scan (`similar` → Similar & unlinked). Both are kept separate from + * `loading` so the note body paints the instant it's read. Each flag drives its + * section's "loading…" hint so an empty section mid-load doesn't read as "nothing found". */ - discovering: boolean; + discoveringSimilar: boolean; + discoveringConnections: boolean; /** The active search query (empty ⇒ the side pane shows discovery, not results). */ searchQuery: string; searchResults: SearchResult[]; @@ -106,7 +108,8 @@ export const state: AppState = { editConflict: false, similar: [], connections: [], - discovering: false, + discoveringSimilar: false, + discoveringConnections: false, searchQuery: "", searchResults: [], linkTarget: null, diff --git a/ui/style.css b/ui/style.css index 5bada4f..d0d4cf8 100644 --- a/ui/style.css +++ b/ui/style.css @@ -743,6 +743,28 @@ body { margin: 4px 0 8px; } +/* Subtle inline spinner for an in-flight side-pane fetch (discovery's whole-vault + `similar` scan) — better than static "Loading…" text, which can read as stuck. */ +.spinner { + display: inline-block; + width: 14px; + height: 14px; + border: 2px solid var(--border); + border-top-color: var(--muted); + border-radius: 50%; + animation: spin 0.7s linear infinite; +} +@keyframes spin { + to { + transform: rotate(360deg); + } +} +@media (prefers-reduced-motion: reduce) { + .spinner { + animation-duration: 1.6s; + } +} + .cards { display: flex; flex-direction: column;