diff --git a/.gitignore b/.gitignore index cbee55d..b01d4fa 100644 --- a/.gitignore +++ b/.gitignore @@ -28,7 +28,7 @@ crates/b2-desktop/gen/ # don't ignore logs/.keep !/logs/.keep -# The retrieval eval's append-only results log (specs/eval-strategy.md): one JSON +# The retrieval eval's append-only results log (the eval harness, crates/b2-embed/evals/): one JSON # line per scored run, machine-local (scores depend on the machine's model/device). crates/b2-embed/evals/results.jsonl diff --git a/CLAUDE.md b/CLAUDE.md index 08d3dbe..eff4406 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,26 +11,23 @@ A personal, local-first Markdown knowledge vault with an AI layer that **surface notes** for you to connect. The Markdown files stay plain and yours; B2 is the intelligence layer over them, not a container around them. This Cargo workspace is the **index engine + its two dumb adapters** — the `b2` CLI and the Tauri desktop app (with the `ui/` frontend); the design lives in -`planning/`. +`docs/design/`. ## Design docs are the source of truth The code is a *projection of the spec*, and comments cite it constantly (e.g. `data-model.md §2`, -`build spec §1.2`, `index-engine.md §6`). Before changing behavior, read the relevant doc — the -schema must satisfy the data model, never the reverse. - -- `planning/invariants.md` — the **invariant register**: the one-page normative list of what must - always be true (cited by id — S2, G2, …). On conflict with any other doc, it wins. -- `planning/vision-and-scope.md` — the *why*: principles, the two design tenets, v1 scope, locked decisions. -- `planning/data-model.md` — the *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary. -- `planning/index-engine.md` + `planning/specs/completed/index-engine-build.md` — the *how*: SQLite (FTS5 + in-process vector scan; see `research/discovery-scan-strategy.md`) projection, table DDL, the build order, data flows. -- `planning/tasks.md` — the working queue (what's done, what's next). **Read this first to know current state.** - Planned-but-unstarted work lives in GitHub Issues; shipped build specs live in `planning/specs/completed/` - (index engine, desktop MVP, async indexing, projection/embedding split, desktop editing, live preview). -- `planning/user-stories.md` — kernel behavior as testable scenarios. -- `planning/specs/eval-strategy.md` — how model quality (the `Embedder` seam) is measured out-of-CI: - the hand-labelled retrieval + discovery evals (BM25-vs-hybrid ablation, note & passage ranks, - `b2 similar`), the chunker-sweep gate, the results log, and how to run/grow it all. +`index-engine.md §6`). Before changing behavior, read the relevant doc — the schema must satisfy the +data model, never the reverse. The three canonical docs live in `docs/design/`: + +- `docs/design/invariants.md` — the **invariant register**: the one-page normative list of what must + always be true, and the source of *why* (cited by id — S2, G2, …). On conflict with any other doc, it wins. +- `docs/design/data-model.md` — the *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary. +- `docs/design/index-engine.md` — the *how*: SQLite (FTS5 + in-process vector scan) projection, table DDL, data flows. + +Planned-but-unstarted work and the backlog live in [GitHub Issues](https://github.com/AlteredCraft/B2/issues); +shipped build history lives in git. Model quality (the `Embedder` seam) is measured out-of-CI by the +eval harness under `crates/b2-embed/evals/` — the hand-labelled retrieval + discovery evals +(BM25-vs-hybrid ablation, note & passage ranks, `b2 similar`), the chunker-sweep gate, and the results log. ## Commands @@ -110,7 +107,7 @@ so Tauri/wry tracing doesn't pollute the file (an explicit `B2_LOG` is honored v ### The core invariant -**`index = a pure projection of (the vault directory)`.** (The full register: `planning/invariants.md`.) Two storage tiers: +**`index = a pure projection of (the vault directory)`.** (The full register: `docs/design/invariants.md`.) Two storage tiers: 1. **The vault directory** — the source of truth. **Markdown is its sole authored subset** — the only format whose bytes B2 may write; non-`.md` files are *resources* (path-keyed peers contributing @@ -187,7 +184,7 @@ adapters wire the real model. - **Flow ① ingest/reindex** (`ingest.rs`) — parse → stamp missing `b2id` (write file) → project notes, chunks (+FTS), embeddings, and the typed `edges` graph. Two-phase so link resolution is independent of file order. It is **two separately-invokable passes** - (`specs/completed/projection-embedding-split.md`): model-free `project_vault` (notes/chunks/FTS/edges) and + (the `project`/`embed` split, #15): model-free `project_vault` (notes/chunks/FTS/edges) and `embed_vault` (fills the DB-derived missing-vector set); `reindex` composes them, and `search` falls back to BM25-only on a projected-but-unembedded vault. - **Flow ② hybrid search** (`search.rs`) — BM25 (`chunks_fts`) ⊕ vector KNN (an exact in-process scan @@ -225,7 +222,7 @@ Vectors live in **plain tables** — `embeddings(chunk_id, vector)` and `note_ce centroid)` — created at **embed time**, not in the base migration: their existence is the "this vault has an embedding space" signal the projected-but-unembedded fallbacks key on. Every distance is computed **in-process** (`embed::l2_sq`, one sequential scan statement; rationale: -`research/discovery-scan-strategy.md`, #38). `meta` records `(embed_model_id, +#38). `meta` records `(embed_model_id, embed_dim)` — the only place a model swap is detectable. The compute **device** folds into this identity: the real embedder tags its recorded `embed_model_id` with the resolved device (CPU stays the bare repo id; a `--features metal` GPU build appends `@metal`, `b2-embed/src/model.rs`), so a diff --git a/Cargo.toml b/Cargo.toml index 699d6ba..2f871ef 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,9 +3,9 @@ resolver = "2" members = ["crates/b2-core", "crates/b2-embed", "crates/b2-cli", "crates/b2-desktop"] # B2 — local-first Markdown PKM with AI connection discovery. -# Design lives in planning/; this workspace is the index engine -# (planning/specs/index-engine-build.md), built step 0→5 against the -# golden-vault fixtures in fixtures/golden-vault/ (planning/data-model.md §8). +# Design lives in docs/design/; this workspace is the index engine +# (docs/design/index-engine.md), built step 0→5 against the +# golden-vault fixtures in fixtures/golden-vault/ (docs/design/data-model.md §8). # Build DEPENDENCIES optimized even in dev/debug builds, while keeping our own `b2-*` # crates at opt-level 0 so the TDD loop and `cargo test -p b2-core` stay fast. Candle's @@ -13,7 +13,7 @@ members = ["crates/b2-core", "crates/b2-embed", "crates/b2-cli", "crates/b2-desk # matmul backend) is ~13× slower unoptimized, so a plain `tauri dev` / `cargo run` # reindex was painfully slow (a 16-chunk embed batch took ~35s instead of ~2.5s). That # mattered doubly for the desktop app: a reindex cancel is only observed at each embed- -# batch boundary (planning/specs/async-indexing.md §3), so a slow batch made the +# batch boundary (docs/design/index-engine.md), so a slow batch made the # **Cancel** button feel stuck. The `"*"` glob optimizes every dependency (targeting # just the candle crates missed the matmul backend and left it ~4× slower); the per- # crate opt-0 overrides below exclude our own crates so their rebuilds stay instant. diff --git a/README.md b/README.md index 171594b..1c293f4 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ A personal, **local-first** knowledge vault — plain Markdown you fully own — explained connections between them yourself. > **Status:** the design is **locked** and the **index engine is built** (`crates/b2-core`: steps 0→5 -> of the [build spec](planning/specs/completed/index-engine-build.md)). The **`b2` CLI over a typed core API** is +> of the index engine). The **`b2` CLI over a typed core API** is > live (`crates/b2-cli`): point B2 at a folder and `reindex` / `search` / `neighbors` / `explain` it > from the terminal, with `--json` for agents. **Semantic search is real** (`crates/b2-embed`: a > candle-backed local embedder behind the one seam; `b2 init` downloads the model into a shared cache; @@ -31,10 +31,10 @@ explained connections between them yourself. > reconciles external edits live. Reindex is a **cancellable background action** — live progress, a Cancel > button, and the UI stays usable while a large vault indexes; projection and embedding are decoupled, so > a cold vault is browsable/keyword-searchable in seconds while embedding streams behind -> ([specs/completed/](planning/specs/completed/)). Run it with `just app` — pick a vault from the +> ([#15](https://github.com/AlteredCraft/B2/issues/15)). Run it with `just app` — pick a vault from the > in-app switcher, or skip straight to one via `B2_VAULT_PATH`. **Next:** file-type support (resources) — > slice 1, inventory & graph, is built; the wider backlog lives in -> [GitHub Issues](https://github.com/AlteredCraft/B2/issues) ([tasks.md](planning/tasks.md)). +> [GitHub Issues](https://github.com/AlteredCraft/B2/issues). ## What B2 is (the north star) @@ -44,12 +44,12 @@ connected yet** — so the structure of your knowledge grows as you link them, i The files stay plain Markdown on your disk, yours forever; B2 is the **intelligence layer over them, not a container around them**. Humans and AI agents are both first-class users. -Full motivation, scope, and locked decisions: **[vision-and-scope.md](planning/vision-and-scope.md)**. +Full motivation, scope, and locked decisions: **[docs/design/invariants.md](docs/design/invariants.md)**. ## How we build it Two architectural tenets shape every decision (full text: -[vision-and-scope.md → Design philosophy](planning/vision-and-scope.md#design-philosophy)): +[docs/design/invariants.md](docs/design/invariants.md)): - **A volatile vault over a disposable index.** Refactor fearlessly — move, split, merge, compress, trim orphans. The index is a pure projection of your vault (drop it, rebuild it identical); @@ -61,7 +61,7 @@ Two architectural tenets shape every decision (full text: …in service of five product non-negotiables — plain-Markdown source of truth · local-first · zero lock-in · AI-native (not bolted-on) · single binary -([vision-and-scope.md → Principles](planning/vision-and-scope.md#principles--non-negotiables)). +([docs/design/invariants.md](docs/design/invariants.md)). ## The docs @@ -75,14 +75,11 @@ and work with a vault in about ten minutes. Then go deeper: | Doc | What it owns | |---|---| -| [invariants.md](planning/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, cited by id. On conflict with any other doc, it wins. | -| [vision-and-scope.md](planning/vision-and-scope.md) | Why B2 exists · principles · **design philosophy** · v1 scope · locked decisions. The canonical *why*. | -| [data-model.md](planning/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. | -| [index-engine.md](planning/index-engine.md) | How the derived index is *built* — SQLite (FTS5 + an in-process vector scan) as a disposable projection. The canonical *how*. | -| [specs/index-engine-build.md](planning/specs/completed/index-engine-build.md) | The build **spec** — precise table DDL, relations, data flows, and the step 0→5 build order. The buildable contract. | -| [specs/completed/desktop-ui-mvp.md](planning/specs/completed/desktop-ui-mvp.md) | The **desktop UI** build spec — Tauri + CodeMirror, the repo layout, the thin-adapter discipline, and the read→discover→link MVP. The first UI adapter. | -| [user-stories.md](planning/user-stories.md) | Kernel behavior as testable scenarios (rename/move, link delete) · link-identity mechanics. | -| [tasks.md](planning/tasks.md) | The working queue — what's done, what's next. | +| [docs/design/invariants.md](docs/design/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, and the source of *why*, cited by id. On conflict with any other doc, it wins. | +| [docs/design/data-model.md](docs/design/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. | +| [docs/design/index-engine.md](docs/design/index-engine.md) | How the derived index is *built* — SQLite (FTS5 + an in-process vector scan) as a disposable projection. The canonical *how*. | + +Planned work and the backlog live in [GitHub Issues](https://github.com/AlteredCraft/B2/issues); shipped build history lives in git. ## Build and run diff --git a/crates/b2-cli/src/main.rs b/crates/b2-cli/src/main.rs index a690c5a..e639e98 100644 --- a/crates/b2-cli/src/main.rs +++ b/crates/b2-cli/src/main.rs @@ -1,4 +1,4 @@ -//! `b2` — the first adapter over the `b2-core` typed API (vision-and-scope, +//! `b2` — the first adapter over the `b2-core` typed API (invariants.md, //! headless-first: "the CLI is the UI before the UI"). It holds **no engine logic**: //! it parses args, picks + injects the embedder, calls the [`Vault`] façade, and //! prints — human-readable by default, or `--json` for agents. @@ -25,7 +25,7 @@ use std::sync::atomic::{AtomicBool, Ordering}; /// Set by the Ctrl-C handler installed for a foreground `reindex`. The embed loop /// reads it at each batch boundary (through the [`ControlFlow`] the progress closure /// returns — the shipped cancel seam) and stops *after* the current batch: a -/// consistent, re-runnable partial index, never a torn write (async-indexing.md §3/§8). +/// consistent, re-runnable partial index, never a torn write (index-engine.md). static CANCEL: AtomicBool = AtomicBool::new(false); /// Map the Ctrl-C flag onto the reindex embed loop's cooperative-cancel signal — @@ -336,7 +336,7 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> { if p.note_chunks == 1 { "" } else { "s" }, ); let _ = std::io::stderr().flush(); - // Stop after this batch if Ctrl-C was pressed (async-indexing.md §3/§8), + // Stop after this batch if Ctrl-C was pressed, // else carry on. The batch is already written above, so a cancel here // never tears a write. cancel_flow() @@ -388,8 +388,8 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> { Command::Status => { // Read-only coverage report: how much of the vault is embedded (semantic // ranking live vs. keyword-only) and whether a background reindex is in - // flight — the companion to backgrounding a slow reindex with `b2 reindex &` - // (async-indexing.md §8). A pure model-free DB read (#26): open with the fake. + // flight — the companion to backgrounding a slow reindex with `b2 reindex &`. + // A pure model-free DB read (#26): open with the fake. let root = cli.vault_or_cwd(); let (vault, _semantic) = open_vault(&root, false)?; let status = vault.embed_status()?; @@ -543,7 +543,7 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> { } } // If nothing points *at* the note, it's an orphan — surfaced, not - // acted on (user-stories.md Story 2; files are only touched when asked). + // acted on (invariants.md; files are only touched when asked). if !view.connections.iter().any(|c| c.direction == "inbound") { println!("No inbound links — this note is an orphan."); } diff --git a/crates/b2-cli/tests/cli.rs b/crates/b2-cli/tests/cli.rs index 22f150a..7f59f16 100644 --- a/crates/b2-cli/tests/cli.rs +++ b/crates/b2-cli/tests/cli.rs @@ -1,6 +1,6 @@ //! CLI-level tests: run the built `b2` binary against a temp copy of the //! golden-vault fixture and assert its output — the "run a command against a -//! fixture, assert the output" surface vision-and-scope names. The binary path is +//! fixture, assert the output" surface invariants.md names. The binary path is //! `CARGO_BIN_EXE_b2`, which cargo provides to integration tests (so no extra test //! harness dependency is needed). The CLI is a dumb adapter over `b2_core::Vault`; //! these prove the wiring + output shape, not engine behavior (that's the façade @@ -38,7 +38,7 @@ fn copy_dir(src: &Path, dst: &Path) { /// Run `b2 ` and capture the result. The suite runs under the fake /// embedder (`B2_EMBEDDER=fake`) so CI never downloads or runs the real model — it -/// proves the wiring + output shape, not model quality (tasks.md testability 4–5). +/// proves the wiring + output shape, not model quality (CLAUDE.md). fn run(args: &[&str]) -> Output { Command::new(env!("CARGO_BIN_EXE_b2")) .env("B2_EMBEDDER", "fake") diff --git a/crates/b2-core/src/add.rs b/crates/b2-core/src/add.rs index ade19c4..cc21917 100644 --- a/crates/b2-core/src/add.rs +++ b/crates/b2-core/src/add.rs @@ -76,7 +76,7 @@ pub fn add_note( /// [`ingest::project_file`] (chunks + FTS + edges, **no embedder**), the same pass /// `Vault::write` runs after a save. The new note's chunks join the DB-derived /// missing-vector set for any later embed/reindex to fill -/// (projection-embedding-split.md §7.2) — and a body-less note has nothing to +/// (index-engine.md) — and a body-less note has nothing to /// embed anyway. Same validation and refusals as [`add_note`]. #[allow(clippy::too_many_arguments)] pub fn create_note( diff --git a/crates/b2-core/src/chunk.rs b/crates/b2-core/src/chunk.rs index 89dd163..32c3cb8 100644 --- a/crates/b2-core/src/chunk.rs +++ b/crates/b2-core/src/chunk.rs @@ -1,5 +1,4 @@ -//! Body chunker — the qmd heuristic (index-engine.md §1; spec -//! planning/specs/completed/qmd-chunker.md, issue #19). +//! Body chunker — the qmd heuristic (index-engine.md §1, issue #19). //! //! Splits a note body into **size-targeted, overlapping, Markdown-aware** chunks //! that each carry a `heading_path` breadcrumb (the H1 › H2 › H3 stack the chunk diff --git a/crates/b2-core/src/db.rs b/crates/b2-core/src/db.rs index 6901e96..5225c3c 100644 --- a/crates/b2-core/src/db.rs +++ b/crates/b2-core/src/db.rs @@ -4,7 +4,7 @@ //! the `b2id ⇄ path` resolver. //! //! Every connection is opened `WAL` + `foreign_keys=ON` per -//! planning/specs/completed/index-engine-build.md §0. Every table here is a derived +//! index-engine.md. Every table here is a derived //! projection of `Markdown` — nothing is a source of truth. //! //! Vectors live in **plain tables** and every distance is computed in-process @@ -36,7 +36,7 @@ use std::time::Duration; /// byte-clean slate; either way the next `reindex` rebuilds everything queried. /// **4** added the `resources` inventory and widened `edges` with resource targets /// (`dst_resource_path`/`embed`/`caption`) — file-type support slice 1 -/// (planning/specs/resources-inventory-graph.md §1). +/// (data-model.md §10). pub const SCHEMA_VERSION: i64 = 4; /// Statements at or over this take the slow-query WARN path (`B2_SLOW_QUERY_MS` @@ -121,8 +121,8 @@ pub fn open(path: &Path) -> Result { ); // execute_batch tolerates the rows PRAGMA journal_mode / mmap_size return. // busy_timeout: WAL allows one writer at a time, and two short-statement - // writers can now legitimately race (a save during the background embed — - // desktop-editing.md §4). A modest wait turns that contention into a few-ms + // writers can now legitimately race (a save during the background embed). + // A modest wait turns that contention into a few-ms // stall instead of an immediate SQLITE_BUSY error. // mmap_size + cache_size: the whole-space vector scans stream ~100+ MB of blob // rows per call on a real vault; under the 2 MB default cache with no mmap that @@ -143,7 +143,7 @@ pub fn open(path: &Path) -> Result { /// Create the schema and stamp `schema_version`. `IF NOT EXISTS` keeps the CREATEs a /// no-op on reopen; a `schema_version` mismatch drops the derived tables first so the /// next `reindex` rebuilds them (the index is disposable). The DDL mirrors -/// planning/specs/completed/index-engine-build.md §1 (the vector tables are created at +/// index-engine.md (the vector tables are created at /// embed time — see [`ensure_embedding_space`]). fn migrate(conn: &Connection) -> Result<()> { // `meta` must exist before we can read the schema version the index was built at. @@ -151,7 +151,7 @@ fn migrate(conn: &Connection) -> Result<()> { "CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);", )?; // Schema-version gate: a mismatch means the derived tables have the wrong shape. - // The index is disposable (vision-and-scope, "volatile vault over a disposable + // The index is disposable (invariants.md, "volatile vault over a disposable // index"), so drop the stale derived tables and clear `meta` — the next `reindex` // rebuilds everything under the new schema. Children first (FKs); dropping `chunks` // takes its FTS triggers with it. @@ -382,7 +382,7 @@ pub fn prune_notes_except( } // --------------------------------------------------------------------------- -// resources (file-type support slice 1 — planning/specs/resources-inventory-graph.md §2) +// resources (file-type support slice 1 — data-model.md §10) // --------------------------------------------------------------------------- /// One resource's projection into `resources`. Borrowed view like [`NoteRow`] — @@ -566,7 +566,7 @@ pub fn prune_resources_except(conn: &Connection, seen: &HashSet) -> Resu /// returned ids (Flow ①). pub fn replace_chunks(conn: &Connection, note_b2id: &str, chunks: &[Chunk]) -> Result> { // Guarded on existence so the model-free projection pass still never *creates* - // the embedding space (projection-embedding-split.md §4). + // the embedding space (index-engine.md). if embedding_space_exists(conn)? { conn.execute( "DELETE FROM note_centroids WHERE note_b2id = ?1", @@ -598,7 +598,7 @@ pub fn replace_chunks(conn: &Connection, note_b2id: &str, chunks: &[Chunk]) -> R // --------------------------------------------------------------------------- // embeddings — the vector tables are created at embed time (not in migrate()): // their *existence* is the "this vault has an embedding space" signal the -// projected-but-unembedded fallbacks key on (projection-embedding-split.md §5). +// projected-but-unembedded fallbacks key on (index-engine.md). // --------------------------------------------------------------------------- /// Whether the embedding space (the `embeddings` table) currently exists. @@ -808,7 +808,7 @@ pub fn embed_progress(conn: &Connection) -> Result<(usize, usize)> { /// Every chunk still lacking a stored vector, as `(note_b2id, path, chunk_id, text)` /// in `(path, seq)` order — the **DB-derived pending set** the embed pass fills -/// (projection-embedding-split.md §2). Deriving it here is what decouples projection +/// (index-engine.md). Deriving it here is what decouples projection /// from embedding: nothing is handed between the two passes in memory, so any stop /// point (a cancelled embed, a crash between the passes) heals on the next embed. /// The ordering reproduces the fused reindex's per-note batching + progress. @@ -868,7 +868,7 @@ pub fn all_notes(conn: &Connection) -> Result Result> = db::note_chunk_vectors(conn, anchor)? diff --git a/crates/b2-core/src/embed.rs b/crates/b2-core/src/embed.rs index 78afd13..ed96810 100644 --- a/crates/b2-core/src/embed.rs +++ b/crates/b2-core/src/embed.rs @@ -1,5 +1,5 @@ -//! The embedder seam (planning/index-engine.md §6; the "build for tomorrow's -//! model" tenet in vision-and-scope). Producing embeddings inside a single binary +//! The embedder seam (index-engine.md §6; the "build for tomorrow's +//! model" tenet in invariants.md). Producing embeddings inside a single binary //! is the one genuinely hard part, and it is *orthogonal* to the store — so it //! sits behind this trait. The engine is built and tested against a deterministic //! fake; the real local model (`b2-embed`'s candle-backed `LocalEmbedder`, @@ -104,7 +104,7 @@ pub fn pack_f32(v: &[f32]) -> Vec { /// Inverse of [`pack_f32`]: read a stored BLOB (little-endian float32, no header) /// back into a vector. Used to reuse a note's *stored* chunk vectors as discovery -/// queries without re-embedding (tasks.md ①). A trailing partial group can't occur +/// queries without re-embedding (index-engine.md §3). A trailing partial group can't occur /// for a vector written by [`pack_f32`], so a non-multiple-of-4 length is simply /// truncated rather than treated as an error. pub fn unpack_f32(bytes: &[u8]) -> Vec { diff --git a/crates/b2-core/src/error.rs b/crates/b2-core/src/error.rs index 0664bfe..a053e88 100644 --- a/crates/b2-core/src/error.rs +++ b/crates/b2-core/src/error.rs @@ -30,7 +30,7 @@ pub enum Error { /// The index's recorded embedding model/dim differs from the active embedder, /// so its vectors are incomparable with new query vectors. A read (search) /// fails fast with this rather than returning silently wrong results; the fix - /// is a `reindex` (which re-embeds). See index-engine.md §8 and tasks.md. + /// is a `reindex` (which re-embeds). See index-engine.md §8 and GitHub Issues. #[error("index built with embedding model {indexed}, but the active model is {active}; run `b2 reindex`")] ModelMismatch { indexed: String, active: String }, @@ -76,7 +76,7 @@ pub enum Error { /// `Vault::write` was handed a `base_revision` that no longer matches the file /// on disk — an external editor changed the note since it was read. Refused - /// rather than clobbered (desktop-editing.md §3): the caller re-reads (getting + /// rather than clobbered: the caller re-reads (getting /// the current revision) and either reloads or knowingly re-writes. The path is /// carried for the debug detail, never for the user-facing message. #[error("write conflict: {0} changed on disk since it was read")] diff --git a/crates/b2-core/src/graph.rs b/crates/b2-core/src/graph.rs index 47aa87c..0e12867 100644 --- a/crates/b2-core/src/graph.rs +++ b/crates/b2-core/src/graph.rs @@ -1,6 +1,6 @@ //! Graph queries over the typed `edges` table. Inversion (backlinks) is the //! reason the graph is materialized rather than parsed at read time -//! (planning/index-engine.md §3): a note's inbound edges live in every *other* +//! (index-engine.md §3): a note's inbound edges live in every *other* //! note, so `neighbors` is one indexed lookup, not a full-vault scan. use crate::error::Result; diff --git a/crates/b2-core/src/ingest.rs b/crates/b2-core/src/ingest.rs index 2a1f0e1..c0dc39f 100644 --- a/crates/b2-core/src/ingest.rs +++ b/crates/b2-core/src/ingest.rs @@ -1,9 +1,9 @@ -//! Ingest (Flow ① of planning/specs/completed/index-engine-build.md): parse → stamp a +//! Ingest (Flow ① of index-engine.md): parse → stamp a //! missing `b2id` (write file) → project into `notes`/`note_aliases`, //! `chunks` (+FTS), and the typed `edges` graph. //! //! A full ingest is **two separately-invokable passes** -//! (planning/specs/completed/projection-embedding-split.md §4): [`project_vault`] — the +//! (index-engine.md): [`project_vault`] — the //! model-free pass, which runs in two phases so link resolution never depends on //! file order (phase 1 projects every note + its chunks, filling the resolver; //! phase 2 derives edges against the now-complete resolver) — and [`embed_vault`] — @@ -35,8 +35,8 @@ use std::path::Path; /// chunk in a batch to the batch's *longest*, so an over-large batch runs the whole /// forward pass at the longest length. Measured on a real (variable-length) vault, 16 /// beat 32 (~40% faster: less padding waste) and 8 (better amortization). It also sets -/// the reindex **cancel granularity**: the cancel flag is checked once per batch -/// (async-indexing.md §3), so a smaller batch means the desktop **Cancel** responds +/// the reindex **cancel granularity**: the cancel flag is checked once per batch, +/// so a smaller batch means the desktop **Cancel** responds /// sooner — another reason not to over-size it. const EMBED_BATCH: usize = 16; @@ -78,7 +78,7 @@ pub struct PlannedNote { /// a handful of notes are doing any work. /// /// `Serialize` so the desktop host can stream it to the webview over a -/// `tauri::ipc::Channel` (async-indexing.md §4); the field names are the JSON keys the +/// `tauri::ipc::Channel`; the field names are the JSON keys the /// frontend reads. #[derive(Debug, Clone, Serialize)] pub struct ReindexProgress { @@ -113,7 +113,7 @@ pub struct ReindexProgress { /// passes `false`: it reads only `notes` (`force || body changed || note is new`), /// because "unchanged body but missing vectors" is [`embed_vault`]'s job, not a /// reason to re-chunk — and this is what keeps [`project_vault`] free of the -/// vector tables (projection-embedding-split.md §4). [`ingest_file`] passes `true` +/// vector tables (index-engine.md). [`ingest_file`] passes `true` /// (it embeds inline and has ensured the space exists), so a note left mid-embed is /// also healed by [`would_reembed`]'s vector-state check, exactly as before. /// @@ -121,7 +121,7 @@ pub struct ReindexProgress { /// the `Vault`, which defaults it) so *every* path that chunks a given vault cuts /// identically and `incremental ≡ full rebuild` holds by construction. The retrieval /// eval injects non-default configs here to A/B chunker levers in one process -/// (specs/eval-strategy.md). +/// (the eval harness, crates/b2-embed/evals/). fn project_note_and_chunks( conn: &Connection, vault_root: &Path, @@ -242,7 +242,7 @@ fn would_reembed( /// at a batch boundary, and whether **every** pending chunk got a vector. struct NoteEmbedOutcome { /// `on_batch` returned [`ControlFlow::Break`] at a batch boundary — the caller - /// should stop starting new notes (a cooperative cancel, async-indexing.md §3). + /// should stop starting new notes (a cooperative cancel). cancelled: bool, /// Every pending chunk was embedded, so the note is now fully embedded. True even /// when the cancel landed on the *final* batch: each batch is written before its @@ -256,7 +256,7 @@ struct NoteEmbedOutcome { /// cancel). Chunk vectors are independent, so batch boundaries never change the result. /// /// The cancel check runs **after** a batch is fully written, so a cancel never tears a -/// batch (async-indexing.md §5.6) — it only stops *further* batches. Returns whether a +/// batch — it only stops *further* batches. Returns whether a /// cancel was seen and whether the note finished embedding (see [`NoteEmbedOutcome`]). /// /// A note that finishes has its **centroid** refreshed from its now-complete stored @@ -491,8 +491,8 @@ pub fn ingest_file( } /// The result of a (possibly cancelled) full ingest: every projected note, plus -/// whether the embed phase was cut short by a cooperative cancel (async-indexing.md -/// §3). A cancelled run is still **consistent** — every note has chunks + FTS + edges +/// whether the embed phase was cut short by a cooperative cancel. A cancelled run is +/// still **consistent** — every note has chunks + FTS + edges /// (Phase 1/2), only a *prefix* has vectors — so `notes` describes the partial work /// truthfully (its `embedded` flags count only notes that fully embedded this run) and /// an incremental re-run embeds the notes the cancel left unfinished. Vectors are @@ -548,7 +548,7 @@ pub struct Projected { /// Re-project a single note at `vault_root/rel_path` **model-free** — the /// single-note sibling of [`project_vault`], and the pass `Vault::write` runs after -/// its body splice (desktop-editing.md §4): note + chunks (+FTS) + edges, stamping +/// its body splice: note + chunks (+FTS) + edges, stamping /// a missing `b2id`, never touching the embedding space. A changed body re-chunks /// (clearing its stale vectors), and the chunks join the DB-derived pending set for /// **any** later embed pass to fill — so the save path needs no embedder and no @@ -624,7 +624,7 @@ pub struct EmbedOutcome { pub cancelled: bool, } -/// The **projection pass** (projection-embedding-split.md §4): project every `.md` +/// The **projection pass** (index-engine.md): project every `.md` /// file under `vault_root` — Phase 1 (note + chunks + FTS, stamping missing /// `b2id`s) then Phase 2 (the typed edges) — with **no embedder and no embedding /// space**: it never creates the vector tables, so it needs neither the model nor @@ -724,14 +724,14 @@ pub fn project_vault( }) } -/// The **embed pass** (projection-embedding-split.md §4): fill a vector for every +/// The **embed pass** (index-engine.md): fill a vector for every /// chunk that lacks one. Ensures the embedding space first (creates the /// `embeddings` + `note_centroids` tables; a model swap drops + resets them, so /// *all* chunks then count as missing), then works the DB-derived pending set /// ([`db::chunks_missing_vectors`]) /// note by note through the batched [`embed_pending`] loop — firing `on_progress` /// per batch and honoring its [`ControlFlow::Break`] as the cooperative cancel -/// checkpoint (async-indexing.md §3). Takes **no `force`**: re-chunking (which +/// checkpoint. Takes **no `force`**: re-chunking (which /// clears vectors) is a projection concern, so this pass is purely "fill what's /// missing" — which is also why any interruption heals on the next call (§7.2). /// @@ -813,10 +813,10 @@ pub fn embed_vault( /// when `on_progress` returns [`ControlFlow::Break`], the embed pass stops at that /// batch boundary. Projection (notes + chunks + FTS **and** edges) has completed /// before embedding starts, so a cancelled index is consistent — keyword search + -/// graph are complete, only a prefix of notes has vectors (async-indexing.md §3/§5). +/// graph are complete, only a prefix of notes has vectors. /// /// A thin composition of [`project_vault`] then [`embed_vault`] -/// (projection-embedding-split.md §4): from a clean index the composed run is +/// (index-engine.md): from a clean index the composed run is /// byte-identical to the old fused one; the sole intentional divergence is a /// resume-after-partial run, where projection leaves an unchanged-body note's /// chunks in place rather than regenerating their rowids — observably identical @@ -913,7 +913,7 @@ pub fn plan_reindex(conn: &Connection, vault_root: &Path, force: bool) -> Result /// Walk the vault once, routing every file: `.md` (case-insensitive) → `notes`, /// everything else → `resources` with its class, per /// [`ResourceClass::of_path`] — the `index = projection of (the vault directory)` -/// walk (planning/specs/resources-inventory-graph.md §2). Dot-prefixed +/// walk (data-model.md §10). Dot-prefixed /// **directories** are skipped as always (`.b2/`, `.git/`); dot-prefixed **files** /// are skipped from the resource inventory (`.DS_Store`, `.gitignore` are not /// vault material) while the note route keeps its historical behavior. diff --git a/crates/b2-core/src/lib.rs b/crates/b2-core/src/lib.rs index 7c09058..1d24b30 100644 --- a/crates/b2-core/src/lib.rs +++ b/crates/b2-core/src/lib.rs @@ -1,7 +1,7 @@ //! B2 index engine (`b2.sqlite`) — a **disposable** projection of `Markdown`. //! -//! Built step 0→5 per `planning/specs/completed/index-engine-build.md`; the schema is a -//! derived projection of `planning/data-model.md` and must satisfy it, never the +//! Built step 0→5 per `index-engine.md`; the schema is a +//! derived projection of `data-model.md` and must satisfy it, never the //! reverse. Step 0 is the substrate: open the DB with the locked pragmas over the //! bundled, statically-linked SQLite (FTS5 compiled in; vectors are plain BLOB //! tables scored in-process since schema v3, #38 — no extension needed). diff --git a/crates/b2-core/src/link.rs b/crates/b2-core/src/link.rs index 5ff3c9e..4f6264b 100644 --- a/crates/b2-core/src/link.rs +++ b/crates/b2-core/src/link.rs @@ -1,5 +1,5 @@ -//! Parse a note body into the links that become edges (planning/data-model.md §2; -//! resource forms: planning/specs/resources-inventory-graph.md §3). +//! Parse a note body into the links that become edges (data-model.md §2; +//! resource forms: data-model.md §10). //! //! **The body carries no B2 syntax** (data-model §2, decision 2026-07-21): every //! body construct is ordinary Obsidian Markdown and yields an untyped diff --git a/crates/b2-core/src/mv.rs b/crates/b2-core/src/mv.rs index d9e69ab..b93d9d4 100644 --- a/crates/b2-core/src/mv.rs +++ b/crates/b2-core/src/mv.rs @@ -1,4 +1,4 @@ -//! Move / rename a note and repair inbound links (user-stories.md Story 1). +//! Move / rename a note and repair inbound links (invariants.md). //! //! The typed graph keys every edge by `b2id`, never by path, so a move **never //! breaks the graph** — the target's `b2id` is untouched and every b2id-keyed edge diff --git a/crates/b2-core/src/note.rs b/crates/b2-core/src/note.rs index 5cb2904..244f551 100644 --- a/crates/b2-core/src/note.rs +++ b/crates/b2-core/src/note.rs @@ -94,7 +94,7 @@ impl ParsedNote { /// frontmatter block. Byte-honest like [`body`](Self::body): the actual bytes, /// not a re-serialization of the parsed [`fields`](Self::fields), so keys B2 /// doesn't model (`b2_relations:`, `aliases:`, custom keys) show as written. Powers - /// the Desktop UI's frontmatter drawer (specs/completed/desktop-ui-mvp.md §4). + /// the Desktop UI's frontmatter drawer (crates/b2-desktop/CLAUDE.md). pub fn frontmatter(&self) -> Option<&str> { self.fm.map(|f| &self.raw[f.content_start..f.content_end]) } @@ -120,7 +120,7 @@ impl ParsedNote { } /// Replace the note's **body** with `new_body`, verbatim — the byte-honest - /// splice behind `Vault::write` (desktop-editing.md §4). Everything up to + /// splice behind `Vault::write`. Everything up to /// `body_start` (the frontmatter block and its fences — every byte, including /// keys B2 doesn't model) is preserved *by construction*; a note with no /// frontmatter is replaced wholesale (its body **is** the file). No newline diff --git a/crates/b2-core/src/relation.rs b/crates/b2-core/src/relation.rs index 4a170d1..104fbd1 100644 --- a/crates/b2-core/src/relation.rs +++ b/crates/b2-core/src/relation.rs @@ -1,9 +1,9 @@ -//! The relation vocabulary (planning/data-model.md §2): a closed three-verb +//! The relation vocabulary (data-model.md §2): a closed three-verb //! stance core — `references` (neutral), `supports` (for), `contradicts` //! (against) — with display-only inverse labels and symmetry, plus a tolerated //! tail kept verbatim. The core encodes the one thing embedding similarity //! cannot infer: stance. It is a relaxable policy, not a structural assumption -//! (vision-and-scope, design philosophy) — adding a verb here is the whole +//! (invariants.md) — adding a verb here is the whole //! change. /// A core relation verb and its display metadata. diff --git a/crates/b2-core/src/resource.rs b/crates/b2-core/src/resource.rs index 98b7970..5ad947e 100644 --- a/crates/b2-core/src/resource.rs +++ b/crates/b2-core/src/resource.rs @@ -1,6 +1,5 @@ //! Resource classification and document-kind dispatch — file-type support -//! slice 1 (planning/specs/resources-inventory-graph.md §1/§4; the taxonomy: -//! research/file-type-support.md §3). +//! slice 1 (data-model.md §10). //! //! Class is decided by **extension only** — deterministic, no content sniffing; //! a mislabeled file degrades gracefully rather than mis-executing. The table is @@ -57,7 +56,7 @@ impl ResourceClass { } /// Which arm of the vault an argument names — the pure dispatch rule locked in -/// research/file-type-support.md §9b #8, kept in core so the CLI and the desktop +/// data-model.md §10, kept in core so the CLI and the desktop /// can never drift on it. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum DocKind { diff --git a/crates/b2-core/src/search.rs b/crates/b2-core/src/search.rs index 32c77a0..c0a1c51 100644 --- a/crates/b2-core/src/search.rs +++ b/crates/b2-core/src/search.rs @@ -1,4 +1,4 @@ -//! Hybrid retrieval (planning/index-engine.md §1, §5; build spec Flow ②). +//! Hybrid retrieval (index-engine.md §1, §5; build spec Flow ②). //! //! BM25 (over `chunks_fts`) and brute-force vector KNN (an in-process scan over the //! stored `embeddings`) are retrieved in parallel and fused with **Reciprocal Rank @@ -94,7 +94,7 @@ fn pool_size(limit: usize) -> usize { /// Keyword-only search: BM25 over `chunks_fts` → top `limit`, resolved to notes — /// the fallback that makes a **projected-but-unembedded** vault searchable -/// (projection-embedding-split.md §5): no query embedding, no model, no vectors. +/// (index-engine.md): no query embedding, no model, no vectors. /// Scores are the RRF of the single BM25 list, so they live on the same scale (and /// sort the same way) as [`hybrid_search`]'s fused scores. pub fn keyword_only_search( diff --git a/crates/b2-core/src/vault.rs b/crates/b2-core/src/vault.rs index d8ac438..7b7d2fa 100644 --- a/crates/b2-core/src/vault.rs +++ b/crates/b2-core/src/vault.rs @@ -1,5 +1,5 @@ -//! The `Vault` façade — B2's one typed core API (vision-and-scope, "the -//! testability stack" point 1). Everything before this exists only as modules the +//! The `Vault` façade — B2's one typed core API (invariants.md). Everything before +//! this exists only as modules the //! integration tests call directly; this is the single entry point the `b2` CLI //! (and future adapters) are the sole clients of. It owns the open connection, the //! embedder, and the id generator, and exposes *only what the shipped commands need* @@ -69,7 +69,7 @@ pub struct Vault { conn: Connection, // Injected through the seam: the CLI wires the real candle model; `open` // defaults to `FakeEmbedder` so the core tests stay deterministic and model-free - // (the "build for tomorrow's model" seam, vision-and-scope). + // (the "build for tomorrow's model" seam, invariants.md). embedder: Box, idgen: UlidGen, // The vault's one chunking policy (chunk.rs, spec §3 D5). Held here — not @@ -80,7 +80,7 @@ pub struct Vault { // cut under the old policy, so a config change must pair with // `project(force)` (as `set_chunk_config`'s doc requires and the eval does). // Defaults to `ChunkConfig::default()`; the retrieval eval is the one client - // that overrides it, to A/B chunker levers in-process (specs/eval-strategy.md). + // that overrides it, to A/B chunker levers in-process (the eval harness, crates/b2-embed/evals/). chunk_config: ChunkConfig, } @@ -88,8 +88,8 @@ pub struct Vault { /// (re)embedded (the rest reused their vectors — incremental), and how many needed /// a `b2id` stamped (B2's one always-allowed write to the vault, data-model.md §1). /// -/// `cancelled` is `true` when a cooperative cancel cut the embed phase short -/// (async-indexing.md §3): the counts then describe the partial work truthfully +/// `cancelled` is `true` when a cooperative cancel cut the embed phase short: +/// the counts then describe the partial work truthfully /// (e.g. "indexed 1000, embedded 240, cancelled") — the index is still consistent /// (keyword + graph complete, a prefix embedded) and an incremental re-run finishes /// the rest. Always `false` for [`reindex`](Vault::reindex) and the CLI, which never @@ -114,7 +114,7 @@ pub struct ReindexReport { } /// What [`project`](Vault::project) did — the model-free half of a reindex -/// (projection-embedding-split.md §4): how many notes were projected and how many +/// (index-engine.md): how many notes were projected and how many /// needed a `b2id` stamped. No embed counts: projection never touches vectors. #[derive(Debug, Clone, PartialEq, Eq, Serialize)] pub struct ProjectReport { @@ -144,8 +144,8 @@ pub struct EmbedReport { } /// The vault's semantic-embedding coverage — how many of its notes are fully -/// embedded — for the honest "N/M embedded" signal (#26, projection-embedding-split.md -/// §5). Model-free: a pure count over the projection, so an adapter can surface +/// embedded — for the honest "N/M embedded" signal (#26, index-engine.md). +/// Model-free: a pure count over the projection, so an adapter can surface /// "keyword-only for now" *precisely* (not just via the binary "is a model installed" /// flag) without loading the model. `embedded == total` (and `total > 0`) means /// semantic ranking is complete; `embedded < total` means [`search`](Vault::search) is @@ -266,7 +266,7 @@ pub struct ExplainView { } /// A note's content + display metadata for a reader — the Desktop UI MVP's left -/// pane (specs/completed/desktop-ui-mvp.md §4), and the **one new façade op** that surface +/// pane (crates/b2-desktop/CLAUDE.md), and the **one new façade op** that surface /// adds. Carries the note's identity, the frontmatter fields worth showing a human, /// and the **raw Markdown body read from disk** (the source of truth, not the index /// projection) so an adapter renders Markdown → HTML itself. A pure read — no @@ -286,10 +286,10 @@ pub struct NoteView { /// fences excluded), or `None` when the note has none. This is the byte-honest /// block, not a re-serialization of the projected fields above — so `b2_relations:` /// and any keys B2 doesn't model show as written. The Desktop UI renders it in a - /// collapsible drawer (specs/completed/desktop-ui-mvp.md §4). + /// collapsible drawer (crates/b2-desktop/CLAUDE.md). pub frontmatter: Option, - /// blake3 of the **raw file bytes** at read time — the save-guard token - /// (desktop-editing.md §3/§4): [`write`](Vault::write) refuses when the file on + /// blake3 of the **raw file bytes** at read time — the save-guard token: + /// [`write`](Vault::write) refuses when the file on /// disk no longer hashes to the revision the edit was based on, so a save can /// never silently clobber an external edit. Whole-file (not just the body), so /// *any* out-of-band change conflicts honestly. @@ -357,7 +357,7 @@ pub struct SearchResult { /// Same retrieval (BM25 ⊕ vector → RRF, keyword-only fallback), but ranked chunks /// are returned as-is instead of deduped up to notes, so a caller can see *which /// passage* matched and at what rank. The client is the out-of-CI retrieval eval -/// (specs/eval-strategy.md): note-rank scoring is blind to sub-note retrieval +/// (the eval harness, crates/b2-embed/evals/): note-rank scoring is blind to sub-note retrieval /// quality — exactly what chunking levers move — so the eval scores passage ranks /// through this view. Carries the chunk's **full text** (not a display snippet): /// the eval anchors passage-containment scoring on it; an adapter wanting a @@ -395,7 +395,7 @@ pub struct SimilarView { /// What [`write`](Vault::write) did: the saved note's vault-relative path and the /// **new revision** (blake3 of the final on-disk bytes) — the token the editor /// chains its next save on, so sequential saves never self-conflict -/// (desktop-editing.md §3, "last save wins — by construction"). +/// ("last save wins — by construction"). #[derive(Debug, Clone, PartialEq, Eq, Serialize)] pub struct WriteReport { pub path: String, @@ -424,7 +424,7 @@ impl Vault { /// to inject the real candle model while tests keep the fake. /// /// `open` **never mutates the embedding space** (the `open()`-time-drop fix, - /// tasks.md / index-engine.md §8): shaping the vector tables and any re-embed happen + /// GitHub Issues / index-engine.md §8): shaping the vector tables and any re-embed happen /// only on `reindex`. That way changing the configured model can never silently /// wipe vectors on the next command — a mismatch is caught, and fixed, at /// `reindex`; `search` fails fast on it (see [`search`](Self::search)). @@ -452,7 +452,7 @@ impl Vault { /// `mv` — cuts with this config, so the index stays self-consistent. The /// client is the out-of-CI retrieval eval, which sweeps chunker levers in one /// process (`set_chunk_config` → `project(force)` → `embed` → score; - /// specs/eval-strategy.md); the shipped adapters never call it. Changing the + /// the eval harness, crates/b2-embed/evals/); the shipped adapters never call it. Changing the /// config does **not** re-chunk by itself — pair it with `project(force)`. pub fn set_chunk_config(&mut self, cfg: ChunkConfig) { self.chunk_config = cfg; @@ -473,8 +473,8 @@ impl Vault { /// the real model shows a live progress line instead of looking frozen; and the /// callback's [`ControlFlow`] return **cooperatively cancels** the embed phase — /// returning [`ControlFlow::Break`] stops embedding at that batch boundary while - /// Phase 2 still completes, leaving a consistent, resumable index - /// (async-indexing.md §3). The desktop host maps a cancel flag to `Break`; the CLI + /// Phase 2 still completes, leaving a consistent, resumable index. + /// The desktop host maps a cancel flag to `Break`; the CLI /// always returns `Continue` (no behavior change for the non-cancel path, which /// stays byte-identical). A cancelled run sets [`ReindexReport::cancelled`]. pub fn reindex_with_progress( @@ -504,7 +504,7 @@ impl Vault { }) } - /// The **projection pass** alone (projection-embedding-split.md §4): re-project + /// The **projection pass** alone (index-engine.md): re-project /// every `.md` note into `notes`/`chunks`(+FTS)/`edges` — stamping missing /// `b2id`s — with **no model and no vector work**. After it returns, the file /// tree lists, notes open, keyword search answers, and the graph resolves; only @@ -688,7 +688,7 @@ impl Vault { } /// Read a note for display (`Vault::read`) — the Desktop UI MVP's left pane and - /// the one new façade op that surface adds (specs/completed/desktop-ui-mvp.md §4). Resolve + /// the one new façade op that surface adds (crates/b2-desktop/CLAUDE.md). Resolve /// `note_ref` (path **or** `b2id`) to its file and return the note's **raw /// Markdown body from disk** (the source of truth, not the index projection) plus /// the frontmatter metadata worth showing a reader. A pure read — no embedding, @@ -723,7 +723,7 @@ impl Vault { }) } - /// Save a note's **body** (`Vault::write`, desktop-editing.md §4) — the editing + /// Save a note's **body** (`Vault::write`) — the editing /// surface's one write op. Markdown-first and **model-free**: validate that the /// file on disk still hashes to `base_revision` (else [`Error::WriteConflict`] — /// an external editor changed it; nothing is written), splice `body` in @@ -736,8 +736,7 @@ impl Vault { /// Returns the **new revision** (hashing the *final* on-disk bytes — a /// missing-`b2id` stamp, the one write beyond the body, is reflected), which the /// editor chains its next save on: sequential saves never self-conflict, and - /// only an external write trips the guard ("last save wins — by construction", - /// desktop-editing.md §3). + /// only an external write trips the guard ("last save wins — by construction"). pub fn write(&self, note_ref: &str, body: &str, base_revision: &str) -> Result { let _op = tracing::debug_span!(target: "b2::vault", "write", note = note_ref).entered(); let b2id = self.resolve_ref(note_ref)?; @@ -878,7 +877,7 @@ impl Vault { /// `limit` *notes*. Results are note-level: chunk hits are deduped to the /// highest-scoring chunk per note, so one note never appears twice. /// - /// **Keyword-first fallback** (projection-embedding-split.md §5): when the + /// **Keyword-first fallback** (index-engine.md): when the /// vector space does not exist yet — a projected-but-unembedded vault — this /// runs BM25-only (no query embedding, no model) instead of returning nothing, /// so a vault is searchable the moment [`project`](Self::project) finishes. @@ -975,7 +974,7 @@ impl Vault { /// (`db::embed_progress`), so an adapter can tell the user semantic ranking is /// *partial* — flag results "keyword-only for now" — rather than silently /// under-ranking while a vault embeds behind the first tree paint - /// (projection-embedding-split.md §5). `embedded == 0` on a projected-but-unembedded + /// (index-engine.md). `embedded == 0` on a projected-but-unembedded /// vault; `embedded == total` (with `total > 0`) once every note has vectors. pub fn embed_status(&self) -> Result { let _op = tracing::debug_span!(target: "b2::vault", "embed_status").entered(); @@ -1136,7 +1135,7 @@ impl Vault { /// Move/rename the note `note_ref` (path **or** `b2id`) to `to` (a /// vault-relative path; a `.md` suffix is optional), rewriting every inbound /// `[[oldpath|alias]]` link to the new path and re-projecting the index - /// (user-stories.md Story 1). The graph never breaks — edges key on `b2id`, so + /// (invariants.md). The graph never breaks — edges key on `b2id`, so /// `neighbors`/backlinks show the same set before and after; only the human /// convenience-copy link text is repaired. Errors with [`Error::NoteNotFound`] /// for an unknown source, or [`Error::MoveDestination`] / @@ -1248,7 +1247,7 @@ impl Vault { /// creation works with no model provisioned and a fake-opened vault can never /// write foreign vectors into a real-model embedding space. The note's chunks /// join the DB-derived missing-vector set, healed by any later - /// [`embed`](Self::embed)/reindex (projection-embedding-split.md §7.2) — and an + /// [`embed`](Self::embed)/reindex (index-engine.md) — and an /// empty body has nothing to embed anyway. Same refusals as `add_note`: /// [`Error::AddDestination`] / [`Error::AddTargetExists`]. pub fn create_note(&self, path: &str) -> Result { @@ -1301,7 +1300,7 @@ impl Vault { } } -/// A file's save-guard revision: blake3 of its raw bytes (desktop-editing.md §3). +/// A file's save-guard revision: blake3 of its raw bytes. /// One tiny fn so `read` (capture) and `write` (validate + return) can never drift. fn revision_of(raw: &str) -> String { blake3::hash(raw.as_bytes()).to_hex().to_string() diff --git a/crates/b2-core/tests/cancel.rs b/crates/b2-core/tests/cancel.rs index f4b112c..cfb048c 100644 --- a/crates/b2-core/tests/cancel.rs +++ b/crates/b2-core/tests/cancel.rs @@ -1,4 +1,4 @@ -//! Cooperative-cancel of a reindex (planning/specs/completed/async-indexing.md §3): the embed +//! Cooperative-cancel of a reindex: the embed //! phase can be stopped at a batch boundary via `ControlFlow::Break`, and the result //! is a **consistent, resumable** index — every note has chunks + FTS + edges (keyword //! search + graph complete), only a *prefix* has vectors, and an incremental re-run diff --git a/crates/b2-core/tests/chunks.rs b/crates/b2-core/tests/chunks.rs index 1d4394f..25572c7 100644 --- a/crates/b2-core/tests/chunks.rs +++ b/crates/b2-core/tests/chunks.rs @@ -1,4 +1,4 @@ -//! The qmd-heuristic chunker (planning/specs/completed/qmd-chunker.md, issue #19): size-targeted, +//! The qmd-heuristic chunker (index-engine.md, issue #19): size-targeted, //! overlapping, Markdown-aware chunks carrying a `heading_path`. Two DB-level tests keep //! the projection wiring honest; the rest exercise `chunk_body` as the pure function it is. @@ -72,7 +72,7 @@ fn fts_index_tracks_chunks_and_matches_body_text() { #[test] fn vault_chunk_config_reaches_projection() { - // The eval's sweep seam (specs/eval-strategy.md): a non-default ChunkConfig + // The eval's sweep seam (the eval harness, crates/b2-embed/evals/): a non-default ChunkConfig // set on the Vault must actually shape the cut — `set_chunk_config` + // `project(force)` on the same vault re-chunks under the new policy, so a // much finer target yields more chunks than the default did. Model-free. diff --git a/crates/b2-core/tests/common/mod.rs b/crates/b2-core/tests/common/mod.rs index 1d55a3d..1830bed 100644 --- a/crates/b2-core/tests/common/mod.rs +++ b/crates/b2-core/tests/common/mod.rs @@ -6,7 +6,7 @@ use std::cell::Cell; use std::fs; use std::path::Path; -/// b2ids of the two golden-vault notes (planning/data-model.md §8). +/// b2ids of the two golden-vault notes (data-model.md §8). pub const MEMORY_ID: &str = "01JMEM0000000000000000000A"; pub const SRS_ID: &str = "01JSRS0000000000000000000B"; diff --git a/crates/b2-core/tests/discover.rs b/crates/b2-core/tests/discover.rs index ad6b383..16ac243 100644 --- a/crates/b2-core/tests/discover.rs +++ b/crates/b2-core/tests/discover.rs @@ -1,4 +1,4 @@ -//! Connection-discovery candidate generation (planning/tasks.md ①, resolved +//! Connection-discovery candidate generation (index-engine.md §3, resolved //! 2026-07-01): candidates are the *complement* of the graph — notes near an anchor //! in vector space but **not** already connected (self + direct neighbors excluded), //! with 2-hop (triadic-closure) notes deliberately kept. diff --git a/crates/b2-core/tests/embed.rs b/crates/b2-core/tests/embed.rs index bba7121..88f2e84 100644 --- a/crates/b2-core/tests/embed.rs +++ b/crates/b2-core/tests/embed.rs @@ -1,5 +1,5 @@ //! Step 3 — the vector store + the embedder seam -//! (planning/specs/completed/index-engine-build.md step 3): a deterministic fake embedder +//! (index-engine.md): a deterministic fake embedder //! produces reproducible KNN; `embed_model_id`/`embed_dim` are recorded; a //! model/dim swap recreates the vector space; note centroids (discovery's coarse //! stage, #38) track the stored chunk vectors. diff --git a/crates/b2-core/tests/frontmatter.rs b/crates/b2-core/tests/frontmatter.rs index df2b20c..01f303e 100644 --- a/crates/b2-core/tests/frontmatter.rs +++ b/crates/b2-core/tests/frontmatter.rs @@ -1,6 +1,6 @@ //! Frontmatter `b2_relations:` — the reader (→ origin=frontmatter edges), the //! surgical `add_relation` editor (lossless), and frontmatter-wins dedup -//! (planning/data-model.md §0, §2, §3). +//! (data-model.md §0, §2, §3). mod common; diff --git a/crates/b2-core/tests/graph.rs b/crates/b2-core/tests/graph.rs index 436dc12..a6f131f 100644 --- a/crates/b2-core/tests/graph.rs +++ b/crates/b2-core/tests/graph.rs @@ -1,5 +1,5 @@ //! Step 2 — the typed graph projection + `neighbors`, and the -//! `incremental ≡ full` invariant (planning/specs/completed/index-engine-build.md step 2). +//! `incremental ≡ full` invariant (index-engine.md). mod common; diff --git a/crates/b2-core/tests/ingest_resolver.rs b/crates/b2-core/tests/ingest_resolver.rs index 1ee2c46..6c18925 100644 --- a/crates/b2-core/tests/ingest_resolver.rs +++ b/crates/b2-core/tests/ingest_resolver.rs @@ -1,7 +1,7 @@ //! Step 1 — ingest into `notes`/`note_aliases` and the `b2id ⇄ path` resolver. //! //! Green-scenario assertions for build-plan step 1 -//! (planning/specs/completed/index-engine-build.md §4): ingest the golden vault, resolve +//! (index-engine.md): ingest the golden vault, resolve //! `memory ⇄ path` both ways, and prove a note missing a `b2id` is stamped on disk //! (B2's one always-allowed write; the id travels in the frontmatter — data-model.md §1). diff --git a/crates/b2-core/tests/links.rs b/crates/b2-core/tests/links.rs index 6391351..c5a611d 100644 --- a/crates/b2-core/tests/links.rs +++ b/crates/b2-core/tests/links.rs @@ -1,4 +1,4 @@ -//! Step 2 — link parsing (planning/data-model.md §2). Pure (no DB), so these pin +//! Step 2 — link parsing (data-model.md §2). Pure (no DB), so these pin //! the classification rules directly: every body link is an untyped `references` //! edge (the body carries no B2 syntax; decision 2026-07-21), and the verb + //! explanation are parsed only from a frontmatter `b2_relations:` entry diff --git a/crates/b2-core/tests/mv.rs b/crates/b2-core/tests/mv.rs index 86d9bc0..66f76b7 100644 --- a/crates/b2-core/tests/mv.rs +++ b/crates/b2-core/tests/mv.rs @@ -1,4 +1,4 @@ -//! `b2 mv` — move/rename a note and repair inbound links (user-stories.md Story 1, +//! `b2 mv` — move/rename a note and repair inbound links (invariants.md, //! the locked invariant "rename keeps every backlink resolving"). Driven through //! the [`Vault`] façade against the golden vault (and a small purpose-built vault //! for prefix-safety), fully deterministic under the FakeEmbedder. diff --git a/crates/b2-core/tests/project_embed.rs b/crates/b2-core/tests/project_embed.rs index 471f5f4..7e1803b 100644 --- a/crates/b2-core/tests/project_embed.rs +++ b/crates/b2-core/tests/project_embed.rs @@ -1,5 +1,4 @@ -//! The projection/embedding split (planning/specs/completed/projection-embedding-split.md §8 -//! Step 1, plus Step 2's keyword-first fallback): `project` alone builds the complete +//! The projection/embedding split (index-engine.md): `project` alone builds the complete //! keyword + graph index with **no** vectors and no embedding space; `embed` fills //! exactly the DB-derived missing vectors; and project→embed is **observably** //! equivalent to the fused `reindex` (counts, chunk text, text→vector, edges — never diff --git a/crates/b2-core/tests/read.rs b/crates/b2-core/tests/read.rs index cfc860d..6182f4a 100644 --- a/crates/b2-core/tests/read.rs +++ b/crates/b2-core/tests/read.rs @@ -1,5 +1,5 @@ //! `Vault::read` — the one façade op the Desktop UI MVP adds -//! (planning/specs/completed/desktop-ui-mvp.md §4). Its contract: resolve a note by path +//! (crates/b2-desktop/CLAUDE.md). Its contract: resolve a note by path //! **or** `b2id`, return the note's raw Markdown body **from disk** (source of //! truth, frontmatter stripped) plus the display metadata. A pure read, model-free //! (FakeEmbedder), against the golden-vault fixture. diff --git a/crates/b2-core/tests/resources.rs b/crates/b2-core/tests/resources.rs index 752b7ab..b01e4a2 100644 --- a/crates/b2-core/tests/resources.rs +++ b/crates/b2-core/tests/resources.rs @@ -1,5 +1,5 @@ //! Resources slice 1 — inventory & graph -//! (planning/specs/resources-inventory-graph.md). +//! (data-model.md §10). //! //! Step 0: the v4 schema — the `resources` table exists, `edges` carries the //! resource-target columns (`dst_resource_path`, `embed`, `caption`), dangling diff --git a/crates/b2-core/tests/search.rs b/crates/b2-core/tests/search.rs index cc29a19..4022d81 100644 --- a/crates/b2-core/tests/search.rs +++ b/crates/b2-core/tests/search.rs @@ -1,4 +1,4 @@ -//! Step 5 — hybrid retrieval (planning/specs/completed/index-engine-build.md step 5): +//! Step 5 — hybrid retrieval (index-engine.md): //! BM25 ⊕ vector → RRF fusion (k=60), resolved to notes, plus the graph-filtered //! vector⨝edge join (index-engine.md §3) — the substrate connection discovery //! runs on. @@ -167,7 +167,7 @@ fn graph_filtered_search_restricts_to_reachable_notes() { #[test] fn search_chunks_exposes_passage_level_hits() { // The sub-note view (`Vault::search_chunks`) the retrieval eval scores passage - // ranks through (specs/eval-strategy.md): same retrieval as `search`, no note + // ranks through (the eval harness, crates/b2-embed/evals/): same retrieval as `search`, no note // dedup, each hit resolved to its note path + heading breadcrumb + the chunk's // FULL text — containment-scorable, unlike `SearchResult`'s display snippet. let tmp = tempfile::TempDir::new().unwrap(); diff --git a/crates/b2-core/tests/substrate.rs b/crates/b2-core/tests/substrate.rs index 9c43043..ff32086 100644 --- a/crates/b2-core/tests/substrate.rs +++ b/crates/b2-core/tests/substrate.rs @@ -1,7 +1,7 @@ //! Step 0 — DB skeleton & the substrate bet. //! //! Green-scenario assertions for build-plan step 0 -//! (planning/specs/completed/index-engine-build.md §4): +//! (index-engine.md): //! - FTS5 is compiled in (the `bundled` SQLite). *(Vectors need no substrate proof //! since schema v3, #38: they are plain BLOB tables scored in-process — the //! `sqlite-vec` half of the original bet was retired with the dependency.)* diff --git a/crates/b2-core/tests/vault.rs b/crates/b2-core/tests/vault.rs index 4410268..59ed350 100644 --- a/crates/b2-core/tests/vault.rs +++ b/crates/b2-core/tests/vault.rs @@ -1,5 +1,5 @@ //! The `Vault` façade — the one typed core API the CLI and tests are clients of -//! (vision-and-scope testability stack, point 1). This slice's contract: +//! (invariants.md). This slice's contract: //! `open` / `reindex` / `neighbors` / `search`, resolving a note by path **or** //! `b2id`, against the golden-vault fixture. Fully deterministic (FakeEmbedder), //! so it proves the plumbing, not model quality. diff --git a/crates/b2-core/tests/write.rs b/crates/b2-core/tests/write.rs index e47f2c9..aecc42d 100644 --- a/crates/b2-core/tests/write.rs +++ b/crates/b2-core/tests/write.rs @@ -1,6 +1,6 @@ -//! `Vault::write` — the editing surface's one write op (desktop-editing.md §8 -//! Step 1): a byte-honest body splice guarded by a content-hash revision, followed -//! by a **model-free** re-projection. The invariants under test (§7): frontmatter +//! `Vault::write` — the editing surface's one write op: a byte-honest body splice +//! guarded by a content-hash revision, followed +//! by a **model-free** re-projection. The invariants under test: frontmatter //! bytes are invariant under save; the revision chain never self-conflicts while //! external writes always conflict; the save path needs no model; and a saved note //! converges to exactly what a full rebuild would produce once an embed pass runs. @@ -130,7 +130,7 @@ fn write_reprojects_keyword_graph_and_clears_stale_vectors() { // …edges were re-derived: the body contributes its one reference, and the // frontmatter `supports` survives untouched — a body save never edits the - // frontmatter home (desktop-editing.md §4, data-model §2). + // frontmatter home (data-model §2). let outbound: Vec<(String, String)> = { let mut s = conn .prepare( diff --git a/crates/b2-desktop/CLAUDE.md b/crates/b2-desktop/CLAUDE.md index 463b15e..2a2de8d 100644 --- a/crates/b2-desktop/CLAUDE.md +++ b/crates/b2-desktop/CLAUDE.md @@ -3,7 +3,7 @@ Guidance for Claude Code (and humans) working in this crate. It **inherits** the workspace rules in the [root CLAUDE.md](../../CLAUDE.md) (idiomatic Rust, error policy, determinism, user-facing-error policy) and **adds** the one rule that defines this crate's existence: **stay a dumb adapter.** The full rationale and -the MVP plan live in [planning/specs/completed/desktop-ui-mvp.md](../../planning/specs/completed/desktop-ui-mvp.md); this file +the read→discover→link→edit MVP shipped (its build history is in git); this file is the enforceable in-crate rule. ## What this crate is @@ -40,7 +40,7 @@ what makes that architecture pay off: - **Inherited tests.** A thin host means the façade's existing suite already covers the behavior; this crate needs only a few per-command tests (args in → right façade call → view out). Logic here would need its own parallel tests that the CLI already has. -- **The promise stays true.** [vision-and-scope.md](../../planning/vision-and-scope.md) says the GUI is "a +- **The promise stays true.** [invariants.md](../../docs/design/invariants.md) (E3) says the GUI is "a second dumb adapter over the same contract, inheriting every test the CLI bought." That is only true while this crate stays dumb. Thinness is not tidiness; it's the load-bearing property. @@ -62,9 +62,9 @@ add a UI concern to `b2-core`, that's the signal you're putting logic in the wro ([`Vault::open_with_embedder`](../b2-core/src/vault.rs)) and fails fast with the "run `b2 init`" message if it's absent. Three write-side ops are deliberately **model-free** and open the fake: `project` — the model-free half of a reindex - ([specs/completed/projection-embedding-split.md](../../planning/specs/completed/projection-embedding-split.md) §6), + ([#15](https://github.com/AlteredCraft/B2/issues/15)), so the first tree paint never waits on a model load — `write_note` — the save path - ([specs/completed/desktop-editing.md](../../planning/specs/completed/desktop-editing.md) §3), so editing works with no + ([#13](https://github.com/AlteredCraft/B2/issues/13)), so editing works with no model provisioned and saved chunks are healed by the trailing background embed — and `create_note` — the tree's New-note action, the same posture as the save path (the new note is projected immediately; its vectors fill on the next embed pass). @@ -90,4 +90,4 @@ add a UI concern to `b2-core`, that's the signal you're putting logic in the wro **Tauri IPC only** — the frontend `invoke`s these commands. This crate runs **no HTTP server**. An HTTP/`serve` transport is a *different, deferred adapter* for a *different need* (remote / browser / -agent-over-HTTP); it does not belong here. See [the spec §1/§9](../../planning/specs/completed/desktop-ui-mvp.md). +agent-over-HTTP); it does not belong here. See [#24](https://github.com/AlteredCraft/B2/issues/24). diff --git a/crates/b2-desktop/Cargo.toml b/crates/b2-desktop/Cargo.toml index a0035d6..3444789 100644 --- a/crates/b2-desktop/Cargo.toml +++ b/crates/b2-desktop/Cargo.toml @@ -6,7 +6,7 @@ rust-version = "1.96" description = "B2 desktop app — the Tauri host, a dumb adapter over the b2-core Vault façade (the GUI sibling of b2-cli)" # The dependency arrow points *one way*: this crate depends on b2-core (+ b2-embed); -# the core never learns about Tauri or the UI (specs/desktop-ui-mvp.md §2). So the +# the core never learns about Tauri or the UI (crates/b2-desktop/CLAUDE.md). So the # fast core suite (`cargo test -p b2-core`) never compiles Tauri/webview deps — the # same discipline that keeps b2-embed's candle deps out of the core. @@ -36,7 +36,7 @@ thiserror = "2.0.18" # vault that was open when it was last closed. Same crate/convention b2-embed uses for # its XDG model cache (`dirs::data_dir()/b2/…`), so both live under one vendor dir. dirs = "6" -# Native filesystem watch for auto-reload on external edits (#14 / desktop-ui-mvp §5). +# Native filesystem watch for auto-reload on external edits (#14 / crates/b2-desktop/CLAUDE.md). # Host-owned infrastructure: the watcher coalesces raw OS events into one debounced # `vault-changed` pulse to the webview, which reconciles via existing façade ops — the # webview itself is granted no filesystem permission (least-privilege, capabilities/). diff --git a/crates/b2-desktop/capabilities/default.json b/crates/b2-desktop/capabilities/default.json index 5561cc8..d8046db 100644 --- a/crates/b2-desktop/capabilities/default.json +++ b/crates/b2-desktop/capabilities/default.json @@ -1,7 +1,7 @@ { "$schema": "../gen/schemas/desktop-schema.json", "identifier": "default", - "description": "B2 desktop's capability set. The frontend only calls this app's own #[tauri::command] handlers (which need no permission entry) plus the core defaults the webview requires — nothing more (least-privilege, specs/completed/desktop-ui-mvp.md §6). No plugin, filesystem, or shell permission is granted to the webview: the app reaches the vault only through the Vault façade in the Rust host, never directly from the webview. The dialog plugin is present but host-only — its native folder picker is driven from the `choose_vault` command in Rust, so the webview still cannot open a dialog itself.", + "description": "B2 desktop's capability set. The frontend only calls this app's own #[tauri::command] handlers (which need no permission entry) plus the core defaults the webview requires — nothing more (least-privilege, crates/b2-desktop/CLAUDE.md). No plugin, filesystem, or shell permission is granted to the webview: the app reaches the vault only through the Vault façade in the Rust host, never directly from the webview. The dialog plugin is present but host-only — its native folder picker is driven from the `choose_vault` command in Rust, so the webview still cannot open a dialog itself.", "windows": ["main"], "permissions": ["core:default"] } diff --git a/crates/b2-desktop/src/commands.rs b/crates/b2-desktop/src/commands.rs index 4732d7b..7922fd5 100644 --- a/crates/b2-desktop/src/commands.rs +++ b/crates/b2-desktop/src/commands.rs @@ -1,5 +1,5 @@ //! The `#[tauri::command]` handlers — B2's IPC surface, and the frontend's mirror of -//! the [`Vault`](b2_core::vault::Vault) façade (specs/completed/desktop-ui-mvp.md §3). Each +//! the [`Vault`](b2_core::vault::Vault) façade (crates/b2-desktop/CLAUDE.md). Each //! handler is **deserialize → call one façade method → serialize**: no branch, no //! loop, no rule. If a handler ever needs one, that logic belongs behind the façade //! in `b2-core` (add a façade op, not host logic) — that is the whole discipline that @@ -165,7 +165,7 @@ pub fn open_resource(state: State<'_, AppState>, path: String) -> Result<(), Cmd .map_err(|e| CmdError::OpenFailed(e.to_string())) } -/// Save a note's body — the editing surface's one write (desktop-editing.md §5). +/// Save a note's body — the editing surface's one write (crates/b2-desktop/CLAUDE.md). /// **Model-free** like `project`: `Vault::write` splices the body and re-projects /// without touching vectors, so this opens the fake vault (no model load; saving /// works with nothing provisioned) and runs **outside** the single-in-flight embed @@ -301,7 +301,7 @@ pub fn link( } /// The **projection pass** — the fast, model-free half of a reindex -/// (projection-embedding-split.md §6). One façade call over the **fake** vault (no +/// (index-engine.md). One façade call over the **fake** vault (no /// model load on the first-paint path), so the moment it returns the file tree can /// repopulate and keyword search answers; `embed` then streams behind it. Fast and /// synchronous-feeling; nothing to stream, nothing to cancel. @@ -309,7 +309,7 @@ pub fn link( /// Deliberately **outside** the single-in-flight reindex slot: the slot exists to /// protect the long, vector-writing embed pass, and a `project` racing a vault /// switch is harmless — it writes the `.b2/` of the root it captured at dispatch, -/// idempotently, never the new vault's (§6 "why leaving `project` outside the slot +/// idempotently, never the new vault's ("why leaving `project` outside the slot /// is safe"). #[tauri::command(async)] pub fn project(state: State<'_, AppState>) -> Result { @@ -317,7 +317,7 @@ pub fn project(state: State<'_, AppState>) -> Result { } /// The **embed pass** — fill the missing vectors as an **observable, cancellable -/// background action** (async-indexing.md §4). Tauri runs the `(async)` body on a +/// background action**. Tauri runs the `(async)` body on a /// worker thread, so the window stays live; progress streams to the webview over /// `on_event` (a typed, per-invocation [`Channel`]), and the closure returns /// `ControlFlow::Break` once the shared cancel flag is set — the one cancel @@ -339,7 +339,7 @@ pub fn embed( /// Ask the in-flight embed to stop at its next batch boundary. Runs on a *different* /// worker thread than `embed`, so it observes/sets the shared flag concurrently; the /// embed closure sees it and breaks cooperatively — no thread-killing, no torn -/// writes (async-indexing.md §4/§5.6). A no-op if nothing is running. +/// writes. A no-op if nothing is running. #[tauri::command(async)] pub fn cancel_reindex(state: State<'_, AppState>) { state.request_reindex_cancel(); @@ -448,8 +448,8 @@ fn embed_impl( state: &AppState, on_event: &Channel, ) -> Result { - // Single-in-flight: refuse a second embed rather than race two writers on one DB - // (async-indexing.md §5.4). The UI also disables the button, so this is rarely hit. + // Single-in-flight: refuse a second embed rather than race two writers on one DB. + // The UI also disables the button, so this is rarely hit. if !state.try_start_reindex() { return Err(CmdError::ReindexInFlight); } @@ -467,7 +467,7 @@ fn embed_impl( .unwrap_or_else(|_| b2_embed::DEFAULT_MODEL.to_string()); // Time the embed pass itself — the clock starts *after* the model load above, so the // recorded total is embedding throughput, not one-time setup. `chunks_done` is - // cumulative, so its last value is this run's chunk count (async-indexing.md §4). + // cumulative, so its last value is this run's chunk count. let start = std::time::Instant::now(); let mut chunks_this_run = 0u64; let report = vault.embed(&mut |p| { @@ -512,8 +512,7 @@ fn vault_info_impl(state: &AppState) -> Result { /// every subsequent command via [`AppState::current_root`]. /// /// **Cancels any in-flight reindex first**, waiting for it to wind down before -/// repointing the root, so a reindex can never keep writing the vault the app has left -/// (async-indexing.md §4/§5.4). +/// repointing the root, so a reindex can never keep writing the vault the app has left. fn set_vault_root_impl(state: &AppState, root: &Path) -> Result { state.cancel_and_wait_for_reindex(); state.set_root(root); @@ -610,7 +609,7 @@ fn set_model_impl(model: &str) -> Result, CmdError> { #[cfg(test)] mod tests { //! Thin command-layer tests: args resolve → the façade is called → a view comes - //! back (specs/completed/desktop-ui-mvp.md §7 — "thinness *is* the test strategy"; the + //! back (crates/b2-desktop/CLAUDE.md — "thinness *is* the test strategy"; the //! façade's own suite covers behavior). Model-free: read-path commands open with //! the fake, and setup reindexes with the fake directly, so no model is needed. @@ -855,7 +854,7 @@ mod tests { assert!(!msg.to_lowercase().contains("sqlite")); } - // --- async-indexing: the host's task-lifecycle bits (§4) ---------------------- + // --- The host's task-lifecycle bits ------------------------------------------- // // Thin host-infrastructure tests: the guard + cancel state machine and // switch-cancels-first, all model-free (no reindex actually runs — the core's own @@ -980,7 +979,7 @@ mod tests { .unwrap(); // …so the stale save is refused with the STABLE message the frontend - // string-matches to drive its conflict bar (desktop-editing.md §5) — keep + // string-matches to drive its conflict bar (crates/b2-desktop/CLAUDE.md) — keep // this assertion in lockstep with ui/src/api.ts. let err = write_note_impl(&state, "concepts/memory", "mine", ¬e.revision).unwrap_err(); assert!(matches!( @@ -1067,7 +1066,7 @@ mod tests { #[test] fn write_note_runs_outside_the_reindex_slot() { // Like `project`, a save is deliberately unguarded by the embed slot - // (desktop-editing.md §5): short, model-free, must not queue behind a + // (crates/b2-desktop/CLAUDE.md): short, model-free, must not queue behind a // long-running background embed. let tmp = tempfile::TempDir::new().unwrap(); let root = tmp.path().join("vault"); @@ -1100,7 +1099,7 @@ mod tests { let state = AppState::new(Some(root)); // Hold the slot (a stand-in for an in-flight embed): `project` is deliberately - // unguarded (projection-embedding-split.md §6) and must still run. + // unguarded (index-engine.md) and must still run. assert!(state.try_start_reindex()); let report = project_impl(&state).unwrap(); assert_eq!(report.indexed, 2); diff --git a/crates/b2-desktop/src/error.rs b/crates/b2-desktop/src/error.rs index 2f56869..045f344 100644 --- a/crates/b2-desktop/src/error.rs +++ b/crates/b2-desktop/src/error.rs @@ -1,6 +1,6 @@ //! The host's error type + the generic, actionable, no-internals-leaked mapping to a //! user-facing string — the desktop mirror of the CLI's `user_message` -//! (specs/completed/desktop-ui-mvp.md §3; the repo-wide logging policy in the parent CLAUDE.md). +//! (crates/b2-desktop/CLAUDE.md; the repo-wide logging policy in the parent CLAUDE.md). //! //! [`CmdError`] **serializes to that string**, so a `#[tauri::command]` returning //! `Result` hands the webview a safe, actionable message and never a @@ -24,8 +24,8 @@ pub enum CmdError { /// point B2 at one. #[error("no vault specified")] VaultRequired, - /// A `reindex` was requested while one was already running (single-in-flight, - /// async-indexing.md §4). The UI disables the button, so this is a belt-and- + /// A `reindex` was requested while one was already running (single-in-flight). + /// The UI disables the button, so this is a belt-and- /// suspenders refusal that reaches the webview only in a race. #[error("a reindex is already running")] ReindexInFlight, diff --git a/crates/b2-desktop/src/main.rs b/crates/b2-desktop/src/main.rs index d85607c..14aed07 100644 --- a/crates/b2-desktop/src/main.rs +++ b/crates/b2-desktop/src/main.rs @@ -1,7 +1,7 @@ //! `b2-desktop` — the Tauri host, B2's **second dumb adapter** over the //! [`Vault`](b2_core::vault::Vault) façade (the GUI sibling of `b2-cli`). It holds //! **no engine logic**: each `#[tauri::command]` deserializes its args, calls one -//! façade method, and serializes the result (specs/completed/desktop-ui-mvp.md §3). The rules +//! façade method, and serializes the result (crates/b2-desktop/CLAUDE.md). The rules //! that keep it a *dumb* adapter live in this crate's charter, `CLAUDE.md`. //! //! Two things this file owns, both mirroring the CLI: @@ -17,8 +17,8 @@ //! * **Embedder wiring** — pure reads open with the deterministic fake; anything //! that embeds a query or writes vectors (`search` / `link` / `embed`) opens the //! real [`LocalEmbedder`] and **fails fast** with "run `b2 init`" if it's absent. -//! `project` — the model-free half of a reindex (projection-embedding-split.md -//! §6) — opens the fake, so the first tree paint never waits on a model load. +//! `project` — the model-free half of a reindex (index-engine.md) — opens the fake, +//! so the first tree paint never waits on a model load. //! `B2_EMBEDDER=fake` forces the fake everywhere (offline/dev mode). // This binary is desktop-only (no mobile entry point), so a plain `main` suffices. @@ -47,7 +47,7 @@ use watch::VaultWatcher; const CANCEL_POLL: Duration = Duration::from_millis(25); /// The host's shared state: the active vault root plus the background-reindex control -/// bits (async-indexing.md §4). Resolved once at startup, then **swappable at runtime** +/// bits. Resolved once at startup, then **swappable at runtime** /// by the in-app vault picker (`choose_vault`) — so the root sits behind a [`Mutex`]. /// Every command still opens its own short-lived [`Vault`] over the *current* root /// (SQLite WAL permits concurrent readers + one writer), the faithful mirror of the CLI @@ -58,7 +58,7 @@ const CANCEL_POLL: Duration = Duration::from_millis(25); /// drives and interrupts* the one façade op stays here; *what* to embed stays in the /// core (the charter's line). `reindex_running` is a single-in-flight guard for the /// long, vector-writing **embed** pass (the fast, model-free `project` command runs -/// outside it by design — projection-embedding-split.md §6); a running embed checks +/// outside it by design — index-engine.md); a running embed checks /// `reindex_cancel` at each batch boundary (via the closure it passes to /// `Vault::embed`) and stops cooperatively when it is set. pub struct AppState { @@ -125,15 +125,15 @@ impl AppState { } /// Signal the running reindex to stop at its next batch boundary (the - /// `cancel_reindex` command). Cooperative — never a thread kill, so no torn writes - /// (async-indexing.md §5.6). A no-op if nothing is running. + /// `cancel_reindex` command). Cooperative — never a thread kill, so no torn writes. + /// A no-op if nothing is running. pub fn request_reindex_cancel(&self) { self.reindex_cancel.store(true, Ordering::SeqCst); } /// Cancel any in-flight reindex and **block until it winds down** — used before a - /// vault switch so a reindex can never keep writing the vault the app has left - /// (async-indexing.md §4/§5.4). Re-asserts the cancel flag on every poll so it wins + /// vault switch so a reindex can never keep writing the vault the app has left. + /// Re-asserts the cancel flag on every poll so it wins /// even against a reindex that armed (cleared) it a moment after starting; returns /// immediately when nothing is running. pub fn cancel_and_wait_for_reindex(&self) { @@ -291,7 +291,7 @@ fn main() { // webview holds no opener permission. .plugin(tauri_plugin_opener::init()) .manage(state) - // Filesystem auto-reload (#14 / desktop-ui-mvp §5): its own managed state so the + // Filesystem auto-reload (#14 / crates/b2-desktop/CLAUDE.md): its own managed state so the // pure `AppState` machine stays free of an OS watch handle. Started below once the // app handle exists, and re-pointed on a vault switch (`choose_vault`). .manage(VaultWatcher::default()) diff --git a/crates/b2-desktop/src/watch.rs b/crates/b2-desktop/src/watch.rs index 476e942..0709c25 100644 --- a/crates/b2-desktop/src/watch.rs +++ b/crates/b2-desktop/src/watch.rs @@ -1,8 +1,8 @@ -//! Native filesystem watch → **auto-reload on external edits** (desktop-ui-mvp.md §5, -//! Step 5, [#14](https://github.com/AlteredCraft/B2/issues/14)). B2's premise is that the +//! Native filesystem watch → **auto-reload on external edits** (crates/b2-desktop/CLAUDE.md, +//! [#14](https://github.com/AlteredCraft/B2/issues/14)). B2's premise is that the //! vault is *also* edited outside the app (Obsidian/vim, a `git pull`), so the window has //! to notice "the files changed under me" and reconcile — replacing the editing spec's -//! "stale until you try to save" conflict bar (desktop-editing.md §5) with live +//! "stale until you try to save" conflict bar (crates/b2-desktop/CLAUDE.md) with live //! reconciliation. The conflict bar remains the fallback for the one case this can't cover //! safely: an external edit to the note you are *actively typing in* (never clobber a live //! buffer). diff --git a/crates/b2-embed/Cargo.toml b/crates/b2-embed/Cargo.toml index 02a5ae0..d1e4264 100644 --- a/crates/b2-embed/Cargo.toml +++ b/crates/b2-embed/Cargo.toml @@ -6,8 +6,8 @@ rust-version = "1.96" description = "B2's real local embedder — candle-backed inference behind the b2-core Embedder seam" # Heavy ML deps (candle + tokenizers) live *only* here, never in b2-core, so the -# core test suite stays fast, deterministic, and model-free (tasks.md testability -# points 4–5). The `b2` binary depends on this crate to inject the real model. +# core test suite stays fast, deterministic, and model-free (CLAUDE.md). The `b2` +# binary depends on this crate to inject the real model. [dependencies] b2-core = { path = "../b2-core" } diff --git a/crates/b2-embed/evals/queries.json b/crates/b2-embed/evals/queries.json index 9d14124..7a93bc5 100644 --- a/crates/b2-embed/evals/queries.json +++ b/crates/b2-embed/evals/queries.json @@ -1,5 +1,5 @@ { - "description": "Hand-labelled semantic-retrieval set (specs/eval-strategy.md §3). Each query avoids the target note's keywords (synonyms/paraphrase) so it measures semantic lift, not lexical overlap. `relevant` lists the vault-relative path(s) that should rank first. The corpus clusters notes into confusable topic groups (coffee, sleep, plants, security, geology, cycling), so ranking WITHIN a cluster is tested, not just across unrelated topics. An optional `passage` is a short verbatim phrase from the target passage: those queries are additionally scored at CHUNK level (does the ranked chunk's text contain the phrase?), which is what makes chunking changes visible to the eval (qmd-chunker.md §7, GH #44). Keep passages verbatim, short, and unique within the corpus.", + "description": "Hand-labelled semantic-retrieval set (the eval harness, crates/b2-embed/evals/). Each query avoids the target note's keywords (synonyms/paraphrase) so it measures semantic lift, not lexical overlap. `relevant` lists the vault-relative path(s) that should rank first. The corpus clusters notes into confusable topic groups (coffee, sleep, plants, security, geology, cycling), so ranking WITHIN a cluster is tested, not just across unrelated topics. An optional `passage` is a short verbatim phrase from the target passage: those queries are additionally scored at CHUNK level (does the ranked chunk's text contain the phrase?), which is what makes chunking changes visible to the eval (index-engine.md, GH #44). Keep passages verbatim, short, and unique within the corpus.", "queries": [ { "query": "how do leaves turn light into food", "relevant": ["photosynthesis.md"] }, { "query": "pedalling a bike up a steep hill", "relevant": ["bicycle.md"] }, diff --git a/crates/b2-embed/evals/similar.json b/crates/b2-embed/evals/similar.json index a2a9729..04c055a 100644 --- a/crates/b2-embed/evals/similar.json +++ b/crates/b2-embed/evals/similar.json @@ -1,5 +1,5 @@ { - "description": "Hand-labelled connection-discovery set (specs/eval-strategy.md §3): for each anchor note, the corpus notes a human would say belong next to it — its topic-cluster mates. Scores `Vault::similar` (discover::candidates — the two-stage centroid shortlist + exact max-sim, GH #38), which the retrieval eval alone does NOT cover: note-to-note discovery is a different task from query-to-note search even over the same stored vectors. A hit = any `expected` path in the top candidates. The corpus is unlinked, so no candidate is excluded by the 1-hop graph rule.", + "description": "Hand-labelled connection-discovery set (the eval harness, crates/b2-embed/evals/): for each anchor note, the corpus notes a human would say belong next to it — its topic-cluster mates. Scores `Vault::similar` (discover::candidates — the two-stage centroid shortlist + exact max-sim, GH #38), which the retrieval eval alone does NOT cover: note-to-note discovery is a different task from query-to-note search even over the same stored vectors. A hit = any `expected` path in the top candidates. The corpus is unlinked, so no candidate is excluded by the 1-hop graph rule.", "anchors": [ { "anchor": "espresso.md", "expected": ["french-press.md", "coffee-roasting.md"] }, { "anchor": "insomnia.md", "expected": ["sleep-hygiene.md", "dreaming.md"] }, diff --git a/crates/b2-embed/examples/eval.rs b/crates/b2-embed/examples/eval.rs index 8559f44..b7bc8a8 100644 --- a/crates/b2-embed/examples/eval.rs +++ b/crates/b2-embed/examples/eval.rs @@ -1,7 +1,7 @@ //! Semantic-retrieval + discovery eval — the "separate, occasional pass" scoring -//! model quality out of CI (specs/eval-strategy.md). It lives as an **example**, +//! model quality out of CI (the eval harness, crates/b2-embed/evals/). It lives as an **example**, //! not a test, so it never runs in the deterministic `cargo test` suite and model -//! quality can never flake CI (vision-and-scope testability point 5). Run it on +//! quality can never flake CI (invariants.md). Run it on //! demand: //! //! ```console @@ -21,7 +21,7 @@ //! 3. **Passage rank** — queries labelled with a verbatim `passage` are also //! scored at **chunk** level (`Vault::search_chunks`): note-rank is blind to //! sub-note retrieval, which is exactly what chunking levers move -//! (qmd-chunker.md §7, GH #44). +//! (index-engine.md, GH #44). //! 4. **Discovery** — `evals/similar.json` anchors score `Vault::similar` (the //! centroid-shortlisted candidate generation, #38), which query-retrieval alone //! does not exercise. @@ -181,7 +181,7 @@ fn run() -> Result> { // ---- Phase 1: projection only → the BM25-only baseline. ------------------ // The vector space does not exist yet, so `search`/`search_chunks` run - // keyword-only (projection-embedding-split.md §5) — the ablation costs nothing + // keyword-only (index-engine.md) — the ablation costs nothing // extra: it is the same vault, paused between the two passes. let report = vault.project(false)?; let bm25 = score_pass(&vault, &set.queries)?; diff --git a/crates/b2-embed/src/lib.rs b/crates/b2-embed/src/lib.rs index 9962351..2ab6cc8 100644 --- a/crates/b2-embed/src/lib.rs +++ b/crates/b2-embed/src/lib.rs @@ -7,7 +7,7 @@ //! run against the deterministic `FakeEmbedder`. The `b2` CLI is the only client //! that wires the real model in. //! -//! Decisions (locked 2026-06-30, tasks.md "Next up"): +//! Decisions (locked 2026-06-30, GitHub Issues): //! - **Runtime = `candle` + `hf-hub`** — pure-Rust inference compiled into the //! binary; no external ONNX runtime to ship. `hf-hub` is the download seam. //! - **Model = a BERT-family sentence embedder**, default **BAAI/bge-base-en-v1.5** diff --git a/docs/architecture.html b/docs/architecture.html index 15347cc..1f11895 100644 --- a/docs/architecture.html +++ b/docs/architecture.html @@ -636,7 +636,7 @@

Grounded in the tests

Committing a connection — b2 link → frontmatterfrontmatter.rs6 Explain (orphan flag) · dry-run previewsexplain · dry_run10 Note-authoring CRUD — add · mv (unit + integration)add · mv · pathspec · note31 - The desktop-editing façade ops — read · write (revision guard) · listread · write · list16 + The desktop editing façade ops — read · write (revision guard) · listread · write · list16 Structured logging is parseable JSONLlogging.rs2 The Vault façade · the b2 CLI (similar/link included)vault.rs · cli.rs36 Embedder config & provisioningb2-embed4 @@ -654,7 +654,7 @@

Grounded in the tests

Not yet built

An honest architecture names its boundaries. Connection discovery is built — it is now b2 similar (surface) + b2 link (commit), local and free; the LLM relator was cut, not - deferred. What remains is tuning, scale, and packaging — tracked in planning/tasks.md, not gaps that + deferred. What remains is tuning, scale, and packaging — tracked in GitHub Issues, not gaps that slipped through.

    @@ -679,8 +679,8 @@

    Not yet built

    ui/, the second dumb adapter over the façade: read → discover → link → edit → reconcile, with async cancellable indexing and fs-watch reconciliation). Source of truth for every claim: crates/{b2-core,b2-embed,b2-cli,b2-desktop}/src/ - and the test suite under each crate's tests/. Design rationale lives in planning/ - (data-model.md · index-engine.md · specs/completed/); the backlog is + and the test suite under each crate's tests/. Design rationale lives in docs/design/ + (data-model.md · index-engine.md · invariants.md); the backlog is GitHub Issues. · Home · Quick start · Indexing pipeline · Connection discovery

    diff --git a/planning/data-model.md b/docs/design/data-model.md similarity index 97% rename from planning/data-model.md rename to docs/design/data-model.md index ac3f1c1..a2f45b1 100644 --- a/planning/data-model.md +++ b/docs/design/data-model.md @@ -12,9 +12,9 @@ status: draft > Defines **what a note is** and **what a connection is**, as the plain-Markdown source of truth — > engine-independent. This is the yardstick the index-engine work measures against: the SQLite schema > in [index-engine.md](index-engine.md) (§3) is a *derived projection* of this model, and must satisfy -> it, not the other way round. Context: [vision-and-scope.md](vision-and-scope.md) (principles, scope, -> locked decisions), [user-stories.md](user-stories.md) (link format & identity, kernel scenarios), -> [tasks.md](tasks.md) (the open pieces this doc closes). +> it, not the other way round. The companion design docs are [invariants.md](invariants.md) (the +> normative register — the *why*) and [index-engine.md](index-engine.md) (the *how*); planned work is +> tracked in [GitHub Issues](https://github.com/AlteredCraft/B2/issues). The model has exactly **two source-of-truth objects**, both plain Markdown: @@ -84,7 +84,7 @@ Settled by one principle plus the locked rule that B2 changes the vault only on links, prose is prose — and no prose shape (a list marker, a leading verb) is ever B2 structure (§2, §7). *(The lone body write is the mechanical repair of an inbound wikilink's path on move — fixing a link the human already wrote, never adding one.)* -- **B2 writes a connection only when you commit one** ([vision-and-scope.md](vision-and-scope.md), +- **B2 writes a connection only when you commit one** ([invariants.md](invariants.md), "Review & trust") — with `b2 link`, or a body link you write yourself. Nothing lands in a note that you didn't ask for; there is no agent proposing edges behind your back. @@ -161,11 +161,11 @@ exactly the body-vs-metadata line §0 draws. - **`b2id`** — durable identity, ULID-style; **namespaced** so it never collides with a user's own `id`, an OKF `id`, or another tool's. The graph keys **every** edge by `b2id`, never by path or title - ([user-stories.md](user-stories.md)). Set once and never changes; survives move, rename, split, and + ([invariants.md](invariants.md)). Set once and never changes; survives move, rename, split, and merge. *This is B2's one always-allowed edit to the vault:* B2 stamps a missing `b2id` **as needed** (on first sight of a note) — no `b2 init` gate, no refusing to index — because durable identity is the anchor everything else keys off and must travel in the file itself (it's what lets an out-of-band move - be repaired, [user-stories.md](user-stories.md)). The stamp *is* the write — it lands in the note's + be repaired, [invariants.md](invariants.md)). The stamp *is* the write — it lands in the note's frontmatter, so identity travels with the file and needs no separate record. - **`type`** — what *kind* of note this is (`note`, `concept`, `source`, `person`, `daily`, …). Controlled-but-extensible; unknown values tolerated. This is the OKF entity discriminator (§5). @@ -390,7 +390,7 @@ requires nor manages it, and it is separate from edges.) ## 5. OKF compatibility (export is a no-op, not a migration) -Build *like* OKF for cheap interop; don't depend on it ([vision-and-scope.md](vision-and-scope.md), +Build *like* OKF for cheap interop; don't depend on it ([invariants.md](invariants.md), "Inspiration"). The model already lines up: - **`type`** is the OKF entity discriminator — already required frontmatter (§1). @@ -408,8 +408,8 @@ Net: "export to OKF" is selecting and re-shaping fields that already exist — a ## 6. Invariants & serialization discipline The model exists to make the three locked invariants -([vision-and-scope.md](vision-and-scope.md)) hold by construction — they are the **"volatile vault over -a disposable index"** tenet ([vision-and-scope.md](vision-and-scope.md#design-philosophy)) made +([invariants.md](invariants.md)) hold by construction — they are the **"volatile vault over +a disposable index"** tenet ([invariants.md](invariants.md)) made mechanical (the full register, cited by id: [invariants.md](invariants.md)): - **Round-trip losslessness** (`parse → serialize → parse` is byte-identical). B2 preserves unknown @@ -419,7 +419,7 @@ mechanical (the full register, cited by id: [invariants.md](invariants.md)): aliases preserved verbatim), (c) appending one typed-link string to frontmatter `b2_relations:` on `b2 link`. **The body is never authored by B2** — (a) and (c) are frontmatter, and (b) only repairs a link the human already wrote. Every other byte is untouched — directly satisfying the - Story-1/Story-2 acceptance criteria ([user-stories.md](user-stories.md)). + Story-1/Story-2 acceptance criteria ([invariants.md](invariants.md)). - **`full-reindex ≡ incremental-update`.** The **index = projection of (the vault directory)**: the edge set is a pure function of a note's Markdown plus the `path → b2id` resolution table. Re-deriving one note ≡ re-deriving the vault for that note's edges; dropping `b2.sqlite` and rebuilding from the @@ -472,7 +472,7 @@ defined, that doc is where they're enforced in the store. ## 8. A golden-vault sketch (for the test harness) The smallest fixture that exercises the whole model — an authored typed edge and a bare reference. (Ties -to the testability stack, [vision-and-scope.md](vision-and-scope.md).) +to the testability stack, [invariants.md](invariants.md).) `concepts/memory.md` ```markdown @@ -565,7 +565,7 @@ connected here. §0–§9 define the **authored** objects — note and edge — whose structure a human (or B2, in frontmatter) writes in Markdown. A real vault also holds **resources**: every non-`.md` file — a PDF, a PNG, a `.csv`, an `.html` clipping. This section defines what a resource *is* in the model; the full findings, taxonomy, -rendering, and build plan live in [research/file-type-support.md](research/file-type-support.md), and +rendering, and build plan are tracked in [GitHub issue #66](https://github.com/AlteredCraft/B2/issues/66), and the schema in [index-engine.md](index-engine.md) §3. A resource is a **peer vault member** — not a lesser one, and not a generalized note. The single @@ -620,4 +620,4 @@ defer-by-default posture (§4). Schema, the per-class extraction step, and the t resource" clauses. Generalizing `notes` to hold resources would staple a caveat onto every invariant, write guarantee, and frontmatter behavior in §0–§9; a distinct `resources` table isolates the different *write* contract instead of threading it through the note rules -([research/file-type-support.md](research/file-type-support.md) §7). +(see [#66](https://github.com/AlteredCraft/B2/issues/66)). diff --git a/planning/index-engine.md b/docs/design/index-engine.md similarity index 89% rename from planning/index-engine.md rename to docs/design/index-engine.md index f5f8163..79d367c 100644 --- a/planning/index-engine.md +++ b/docs/design/index-engine.md @@ -9,11 +9,11 @@ status: draft # B2 — Index Engine: rebuild qmd on SQLite -> **Findings for the "Index-engine evaluation" task** ([tasks.md](tasks.md)). Evaluates the idea of -> rebuilding [tobi/qmd](https://github.com/tobi/qmd) on our own SQLite store (FTS5 + an in-process -> vector scan, reranker as a fast follow) instead of adopting qmd as a dependency. Context: -> [vision-and-scope.md](vision-and-scope.md) (semantic search is **engine-gated**; single-binary; -> local-first) and the data model leans in [tasks.md](tasks.md). +> **The engine design — the *how*.** Evaluates rebuilding [tobi/qmd](https://github.com/tobi/qmd) on +> our own SQLite store (FTS5 + an in-process vector scan, reranker as a fast follow) instead of +> adopting qmd as a dependency, and specifies the result. Companion design docs: +> [invariants.md](invariants.md) (the *why*) and [data-model.md](data-model.md) (the *what*); semantic +> search is **engine-gated**, single-binary, local-first. ## TL;DR / recommendation @@ -22,7 +22,7 @@ status: draft - qmd is an excellent *blueprint* for hybrid retrieval (BM25 + vector + RRF + LLM rerank) and proves the whole pipeline runs locally. But it is a **search engine**, and B2 is not — B2 is a **typed graph with hybrid retrieval over it**. qmd has no notion of typed edges, backlinks, or `b2id`-stable identity, - which are the reasons B2 exists ([vision-and-scope.md](vision-and-scope.md), capability areas 3, 5). + which are the reasons B2 exists ([invariants.md](invariants.md)). - SQLite gives us **one embedded store for every *queryable* concern at once** — full-text (FTS5), vectors (plain tables scored in-process), and the typed graph — with transactional consistency across them, so `b2 similar` candidate generation joins all three in a single query. That single-store property is @@ -30,12 +30,11 @@ status: draft cache**: `index = projection of (the vault directory)` — drop it, reindex, get it back identical, with **no durable B2-derived state outside your notes** (two tiers, [data-model.md](data-model.md)). - Because the engine **does** provide vector search, the locked **engine-gated** decision resolves in - favour of **semantic search in v1**, not as a fast follow ([vision-and-scope.md](vision-and-scope.md), - "Decisions locked 2026-06-28"). + favour of **semantic search in v1**, not as a fast follow ([invariants.md](invariants.md)). - The **reranker is a clean fast-follow**: a swappable seam after RRF fusion, exactly as the testability stack wants the AI parts isolated. Retrieval quality is good without it; it's pure upside later. - The one genuinely hard part is **not the engine** — it's **producing embeddings inside a single - binary** ([vision-and-scope.md](vision-and-scope.md), principle #5). qmd solves this with + binary** ([invariants.md](invariants.md)). qmd solves this with `node-llama-cpp` + GGUF + Node 22, a heavy stack that fights the single-binary goal. This is the real decision to make, and it is **orthogonal to choosing SQLite** (see §7). @@ -51,7 +50,7 @@ A local CLI search engine for Markdown, all on-device. The shape worth stealing: - **Chunking:** ~900-token chunks, ~15% overlap, Markdown-aware break-point scoring (H1=100, H2=90, code-fence=80, … blank-line=20, list-item=5), with a 200-token backward scan and quadratic distance decay to pick the cleanest boundary. Optional tree-sitter AST chunking for code files. - **Implemented in B2** (`chunk.rs`, #19 / [specs/completed/qmd-chunker.md](specs/completed/qmd-chunker.md), 2026-07-13), + **Implemented in B2** (`chunk.rs`, #19, 2026-07-13), with four model-free adaptations: a **~450**-token target (headroom under bge's 512 truncation), a `chars/4` proxy for token sizing (the core stays tokenizer-free), an unconditional `heading_path` breadcrumb, and every lever on a `ChunkConfig`. Tree-sitter code chunking stays deferred (#41 / spec §8). @@ -78,7 +77,7 @@ It's a clean, well-thought-out design. The disagreement is **scope**, not qualit | Rerank | ✅ cross-encoder | ✅ fast-follow | | **Typed graph** (`b2id→b2id` edges with a relation type) | ❌ none | ⭐ core (areas 3, 5) | | **Backlinks** (who points at X, typed, over the whole vault) | ❌ none | ⭐ core (area 5) | -| **`b2id`-keyed identity** surviving move/rename | ❌ path-keyed, cache is disposable | ⭐ core (user-stories 1–2) | +| **`b2id`-keyed identity** surviving move/rename | ❌ path-keyed, cache is disposable | ⭐ core (invariants L1) | | **Markdown as source of truth** (index is rebuildable/derived) | ~ index *is* the artifact | ⭐ non-negotiable (principle #1) | | Distribution | npm package, Node runtime | ⭐ single binary (principle #5) | @@ -99,7 +98,7 @@ agent-output discipline, and the MCP surface idea. **What we discard:** the npm/ ## 3. The storage architecture (one disposable SQLite index) One artifact, per the two-tier model ([data-model.md](data-model.md)) and realizing the **"volatile vault -over a disposable index"** tenet ([vision-and-scope.md](vision-and-scope.md#design-philosophy)): a +over a disposable index"** tenet ([invariants.md](invariants.md)): a **disposable** SQLite index holding every queryable concern transactionally. The whole index is **rebuildable from the vault** — drop `b2.sqlite`, re-scan the vault, get back an identical index (the locked `full-reindex ≡ incremental-update` invariant). The vault is the single source of truth (with @@ -107,8 +106,8 @@ Markdown its sole authored subset — notes + every committed edge); the index i **no durable B2-derived state outside your notes**. > The precise DDL, the relations between these tables, the read/write data flows, and the build order -> are specified in **[specs/completed/index-engine-build.md](specs/completed/index-engine-build.md)**. The sketch below is -> the orientation; that doc is the buildable contract. +> are realized in the code (`crates/b2-core/src/db.rs` schema + `ingest.rs` flows). The sketch below is +> the orientation; the code is the buildable contract. ``` b2.sqlite — DISPOSABLE CACHE (= projection of Markdown; drop & rebuild any time) @@ -134,12 +133,12 @@ Every table is derived from the vault; there is no third home. *(The projection is built in two separately-invokable passes — model-free `project` (notes/chunks/FTS/edges) then `embed` (vectors), with `reindex` their composition — so keyword search + graph are usable before embedding completes; -[specs/completed/projection-embedding-split.md](specs/completed/projection-embedding-split.md). The invariant is untouched: +the `project`/`embed` split ([#15](https://github.com/AlteredCraft/B2/issues/15)). The invariant is untouched: a projected-but-unembedded index is a smaller projection, never a wrong one.)* **Resources widen the projection.** A real vault also holds non-`.md` files, and the walk inventories them. The locked -design ([data-model.md](data-model.md) §10, [research/file-type-support.md](research/file-type-support.md)) +design ([data-model.md](data-model.md) §10, [#66](https://github.com/AlteredCraft/B2/issues/66)) adds them as **path-keyed peers** without disturbing any statement above — the source *tier* is the whole vault directory, so **`index = projection of (the vault directory)`**: @@ -150,7 +149,7 @@ whole vault directory, so **`index = projection of (the vault directory)`**: same three questions — what index text, can it be a graph endpoint, how does it render. - **`chunks` generalizes** from `note_b2id` to a **document reference** (a note `b2id` *or* a resource path — as one-of nullable FKs on the single table, CASCADE intact for both parents; locked, - [research/file-type-support.md](research/file-type-support.md) §9b #7); search resolves hits up to the + [#66](https://github.com/AlteredCraft/B2/issues/66)); search resolves hits up to the owning document and results carry a `kind`. **Centroids follow** — two-stage discovery's coarse stage scans only centroids (#38, §4 update), so a resource with chunks but no centroid would be searchable yet invisible to `b2 similar`; a sibling `resource_centroids` table (same locked call) is maintained through @@ -165,14 +164,14 @@ whole vault directory, so **`index = projection of (the vault directory)`**: `![[file.ext]]` embed, capturing the alt/caption text on the edge (it becomes the image's index text). - **No migration, ever.** Because the index is disposable this is a `schema_version` bump + rebuild — the disposable-index tenet paying rent. The `resources` DDL lands in the **slice-1 build spec** - ([tasks.md](tasks.md)); the chunk/centroid generalization and the per-class extraction step land in + ([#65](https://github.com/AlteredCraft/B2/issues/65)); the chunk/centroid generalization and the per-class extraction step land in slice 3's; the PDF text-extraction *dependency* (which crate, and its home) is deferred to slice 4 by design. Why this shape fits B2 specifically: - **Edges key on `b2id`, never path** — directly implements the link-identity decision - ([user-stories.md](user-stories.md), [data-model.md](data-model.md)). `notes.b2id` is the durable + ([data-model.md](data-model.md) §1). `notes.b2id` is the durable frontmatter identity (B2's one always-allowed write); `src_id`/`dst_id` and `note_b2id` all hold `b2id` values. A move rewrites `notes.path` and inbound `[[path|title]]` text; every row in `edges` is untouched because it never referenced the path. "Rename keeps every backlink resolving" becomes a @@ -203,7 +202,7 @@ edges is what turns the following from full-vault scans (or impossibilities) int - **Backlinks / inversion.** "Who points at X" cannot be read from X — only from every *other* note. The runtime answer is O(vault) per query; the table makes it one lookup. This is also what services - *"rename keeps every backlink resolving"* ([user-stories.md](user-stories.md), Story 1): the edges name + *"rename keeps every backlink resolving"* ([invariants.md](invariants.md) L1): the edges name the exact N inbound files to rewrite on a move instead of scanning the vault to find them (§8). - **Typed multi-hop traversal.** "notes within 2 hops of X via `supports`/`contradicts`" is a scan *per hop* at runtime; over `edges` it is one SQL traversal. @@ -211,7 +210,7 @@ edges is what turns the following from full-vault scans (or impossibilities) int join `embeddings ⨝ chunks ⨝ edges`, not expressible as a per-note parse. It is a **scoped-traversal** primitive (search *within* an already-related neighborhood). **`b2 similar`'s candidate generation is its *complement*, not this join:** notes semantically near an anchor but *not* within 1 hop — the links you - *haven't* made (resolved 2026-07-01, see [tasks.md](tasks.md) ①) — where the materialized graph supplies + *haven't* made (resolved 2026-07-01, §3) — where the materialized graph supplies the "∖ already-connected" exclusion. Both stand on the same reason the graph and search indexes must live in **one** store (§2): area-5 discovery is the substrate this enables. @@ -224,7 +223,7 @@ traversable graph is the value-add, not the search. The standing cost of carryin `b2id`-under-`[[path]]` write-amplification budgeted in §8. FTS5 is built into SQLite (BM25 ranking included); vectors need no extension — plain tables scored -in-process ([research/discovery-scan-strategy.md](research/discovery-scan-strategy.md)). Both are +in-process ([#38](https://github.com/AlteredCraft/B2/issues/38)). Both are battle-tested at personal-vault scale. ## 4. Semantic search & the engine-gated decision → verdict @@ -239,8 +238,7 @@ How it runs — an **exact, in-process scan**, no vector extension, no ANN: `note_centroids` — read with one sequential statement and scored in-process (`embed::l2_sq`). A `vec0`-style virtual table charges a per-row shadow-table probe on every scan, which dominates at real-vault scale; the plain-table scan does not. Full analysis + options: - [research/discovery-scan-strategy.md](research/discovery-scan-strategy.md) - ([#38](https://github.com/AlteredCraft/B2/issues/38)). + [#38](https://github.com/AlteredCraft/B2/issues/38). - **Discovery is two-stage:** an O(notes) coarse scan over centroids shortlists candidates, then an exact max-sim rescore over only the shortlist's chunk vectors. - **Does brute force scale to B2?** Yes, comfortably. A personal vault of, say, 10k notes → ~50–100k @@ -265,16 +263,16 @@ Slot it exactly where qmd puts it: **after RRF fusion, before final ranking**, b - This is why the reranker is genuinely deferrable with no architectural debt: it changes *ordering*, not the store, the schema, or the candidate set. "Eventually add a reranker" is a one-stage insertion, not a redesign. It is also **store-agnostic** — a model-side seam above the index, not a property of it, - so no vector-store choice simplifies or blocks it ([research/vector-store-alternatives.md](research/vector-store-alternatives.md) §5). + so no vector-store choice simplifies or blocks it ([#67](https://github.com/AlteredCraft/B2/issues/67)). **Scope — this reranks `b2 search`, not `b2 similar`.** The seam signature `(query, candidates) → scores` is the tell: it needs *query text*, so it reorders **query search** (`b2 search`). **`b2 similar` has no -query** — it is passage↔passage KNN, "near ∖ connected" (§3, [tasks.md](tasks.md) ①) — so this reranker +query** — it is passage↔passage KNN, "near ∖ connected" (§3) — so this reranker does **not** apply to it; the discovery-side ranking levers are the qmd chunker upgrade ([#19](https://github.com/AlteredCraft/B2/issues/19)) and distance-weighting ([#20](https://github.com/AlteredCraft/B2/issues/20)), not this. -**Gate the decision on the eval, not intuition** ([specs/eval-strategy.md](specs/eval-strategy.md)). RRF +**Gate the decision on the eval, not intuition** (the eval harness under `crates/b2-embed/evals/`). RRF is a strong baseline; the reranker buys **top-k precision**, whose value *grows with vault size* (semantic near-misses crowd the top past ~1k notes) and is *highest when an agent consumes top-1/top-3 without a human eye* (the `serve` adapter, [#24](https://github.com/AlteredCraft/B2/issues/24)). Vault size changes @@ -293,7 +291,7 @@ somewhere. qmd's answer is `node-llama-cpp` + auto-downloaded GGUF models + Node 22/Bun, needing ~300 MB–3 GB of model files and a JS runtime. That directly tensions B2's single-binary, no-install-ritual goal -([vision-and-scope.md](vision-and-scope.md), principle #5). Options, roughly in order of single-binary +([invariants.md](invariants.md)). Options, roughly in order of single-binary friendliness: 1. **Bundle a small embedding model + a `llama.cpp`/GGUF runtime, statically linked.** Self-contained, @@ -308,7 +306,7 @@ friendliness: **Recommendation:** make the **embedder a seam** (we need it for tests regardless — and a swappable model seam *is* the **"build for tomorrow's model"** tenet in practice, -[vision-and-scope.md](vision-and-scope.md#design-philosophy)), ship a **local model as the default** +[invariants.md](invariants.md)), ship a **local model as the default** (option 1 or 2), and decide model-download-on-first-run vs. bundled-in-binary as a packaging detail later. Crucially, **none of this blocks the engine work**: build the SQLite store + FTS5 + the vector tables + the typed graph now against the deterministic fake embedder; drop the real @@ -321,7 +319,7 @@ binary — no external ONNX Runtime to ship; `hf-hub` is the download seam). Mod (`~/.local/share/b2/models/`), never a surprise mid-command download; `reindex`/`search` fail fast with "run `b2 init`" if it's absent. **The model source is configurable** (default = an HF repo id; overridable to a mirror, another repo, or a local path for offline installs) via a global TOML at -`$XDG_CONFIG_HOME/b2/config.toml`. Build/execution plan in [tasks.md](tasks.md) "Next up". +`$XDG_CONFIG_HOME/b2/config.toml`. Build/execution plan tracked in [GitHub Issues](https://github.com/AlteredCraft/B2/issues). **Built (2026-07-01).** Shipped in the **`b2-embed`** crate (`LocalEmbedder` behind the `b2-core` `Embedder` seam; candle + `hf-hub`; CLS-pool + L2-normalize; asymmetric query prefix). **Model default @@ -361,7 +359,7 @@ the fast suite. Eval is a `cargo run -p b2-embed --example eval` pass (precision The graph buys B2 its reason to exist (typed, `b2id`-stable edges — §2), but the decision to keep links written as human-clickable `[[path|title]]` while the graph keys on `b2id` -([user-stories.md](user-stories.md), "Link format & identity") has standing operational costs. These are +([data-model.md](data-model.md) §9) has standing operational costs. These are *the trade working as designed*, not defects — but they must be budgeted, tested, and watched. - **Write amplification on move.** The inline `path` is a repairable convenience copy, so moving one note @@ -389,14 +387,14 @@ keep links written as human-clickable `[[path|title]]` while the graph keys on ` rows whose file was skipped as unreadable — the walk saw that file, its `b2id` is merely unknowable this run, so evicting it would lie. Single-note ingest (`add`/`mv`/`write`) touches one note and never prunes. *(Resources churn more than notes — images/PDFs get added and deleted freely — and their - inventory pass prunes the same way; [research/file-type-support.md](research/file-type-support.md) §8.)* + inventory pass prunes the same way; [#66](https://github.com/AlteredCraft/B2/issues/66).)* - **A single unreadable file never fails the whole index.** A real vault holds the odd non-UTF-8 or permission-denied `.md`; projection **skips** it (reported as a `skipped` entry carrying a short, file-level reason, surfaced by the CLI and the desktop) and indexes everything else, rather than aborting the reindex on one file it cannot read. - **Derived-index consistency is a permanent invariant, not a one-time build.** The index is a derived projection of `Markdown` and must never drift from it. Three locked invariants are the tripwires - ([vision-and-scope.md](vision-and-scope.md); the full register: [invariants.md](invariants.md)): + (the full register: [invariants.md](invariants.md)): round-trip losslessness (`parse → serialize → parse`), `full-reindex ≡ incremental-update`, and `rename keeps every backlink resolving`. Every edit path (kernel `b2 mv`, link delete, out-of-band reindex) has to preserve all three or the graph silently @@ -404,7 +402,7 @@ keep links written as human-clickable `[[path|title]]` while the graph keys on ` - **Committed edges are only ever authored, never inferred.** B2 writes an edge only on your command (`b2 link`, or a body link you write) — there is no agent proposing edges and no review queue to keep consistent. Editing the vault can strand a connection — e.g. deleting an authored `A→B` link - ([user-stories.md](user-stories.md), Story 2) — but B2 only ever *surfaces* the consequence (an orphan + ([invariants.md](invariants.md) W4) — but B2 only ever *surfaces* the consequence (an orphan flag in `b2 explain`), never silently rewrites an inbound file or an edge. Files are touched only when asked. ## 9. Recommendation diff --git a/planning/invariants.md b/docs/design/invariants.md similarity index 90% rename from planning/invariants.md rename to docs/design/invariants.md index 2459d8c..3993a35 100644 --- a/planning/invariants.md +++ b/docs/design/invariants.md @@ -9,16 +9,16 @@ status: active # B2 — Invariants > The normative register of what must always be true of B2. Each entry is one testable/reviewable -> claim; the linked doc holds the elaboration and rationale. Product non-negotiables (local-first, -> zero lock-in, …) stay in [vision-and-scope.md](vision-and-scope.md) — this page is the *engine and -> data* contract. +> claim; the linked doc holds the elaboration and rationale. This page is the top of the design set — +> the *why* — with the *what* in [data-model.md](data-model.md) and the *how* in +> [index-engine.md](index-engine.md); product non-negotiables (local-first, zero lock-in, +> single-binary) are captured as invariants here. > > **On conflict, this page wins and the other doc gets fixed.** Changing this page is a deliberate > decision, never a drive-by edit. Cite entries by id (S2, G2, …). The register is the two design tenets — *a volatile vault over a disposable index* and *build for -tomorrow's model* ([vision-and-scope.md → Design philosophy](vision-and-scope.md#design-philosophy)) — -made mechanical. +tomorrow's model* — made mechanical. ## S — Storage: two tiers, one projection @@ -56,8 +56,7 @@ made mechanical. create/move/delete of notes, resources, and folders on explicit command. - **W4 — B2 never deletes, moves, or archives vault files of its own accord.** Consequences of human edits (orphans, dangling links, hash-matched move candidates) are *surfaced*, flagged, or proposed — - never silently applied. ([user-stories.md](user-stories.md) Story 2, - [index-engine.md](index-engine.md) §8) + never silently applied. ([index-engine.md](index-engine.md) §8) - **W5 — Round-trip losslessness.** `parse → serialize → parse` is byte-identical outside the specific edit performed; unknown frontmatter keys survive verbatim, in order. B2's own keys are namespaced (`b2id`, `b2_relations`) so they can never collide; a generic `relations:` key is *not* read. @@ -67,8 +66,7 @@ made mechanical. - **L1 — The graph keys every edge by `b2id`, never by path or title.** The inline `[[path|alias]]` is a repairable convenience copy. Consequence, also locked: **rename keeps every backlink - resolving** — a move rewrites path *text* and zero edge rows. ([user-stories.md](user-stories.md) - "Link format & identity") + resolving** — a move rewrites path *text* and zero edge rows. ([data-model.md](data-model.md) §1) - **L2 — A note's title is its filename.** The frontmatter `title:` key is recognized but inert — round-tripped, never driving display, aliases, or search. `b2 link` therefore writes a bare `[[path]]`, no alias. ([data-model.md](data-model.md) §1, §9) @@ -113,12 +111,11 @@ made mechanical. deterministic, content-addressed fake; a real model drops in through the seam with **no schema or flow change**. Model-compensating machinery (per-pair adjudication, query expansion, heavy orchestration) is deferred or off by default — the Bitter-Lesson tenet. A reranker, if it lands, is - the next seam, not an exception. ([vision-and-scope.md](vision-and-scope.md) "Design philosophy", - [index-engine.md](index-engine.md) §5–§6) + the next seam, not an exception. ([index-engine.md](index-engine.md) §5–§6) - **M2 — The embedding space has one recorded identity: `meta.(embed_model_id, embed_dim)` — and the compute device folds into it** (a Metal build tags the id `@metal`). Any identity change is a model swap: `search` **fails fast** rather than mixing spaces, `reindex` drops and re-embeds, and `open` - **never** mutates the vector space. ([CLAUDE.md](../CLAUDE.md) "Embedding-space discipline", GH #40) + **never** mutates the vector space. ([CLAUDE.md](../../CLAUDE.md) "Embedding-space discipline", GH #40) - **M3 — One embedding space in v1.** Every vault member funnels to *text* through the same model; multimodal spaces and describers are documented future seams, default-off. ([data-model.md](data-model.md) §10) @@ -126,20 +123,20 @@ made mechanical. vector tables are created at embed time, so "tables exist" = "this vault has an embedding space" — the fallbacks (BM25-only search on a projected-but-unembedded vault) key on it. Centroids are derived data sharing the vectors' lifecycle — refreshed by the embed pass, dropped on re-chunk, no - separate invalidation. ([CLAUDE.md](../CLAUDE.md), #38) + separate invalidation. ([CLAUDE.md](../../CLAUDE.md), #38) ## E — Engineering discipline (what keeps the above true) - **E1 — The core is deterministic.** No wall-clock and no randomness inside `b2-core`; ids and timestamps are injected (`IdGen`, `created` params). Clocks and log subscribers live in the - adapters. ([CLAUDE.md](../CLAUDE.md) Conventions) + adapters. ([CLAUDE.md](../../CLAUDE.md) Conventions) - **E2 — `cargo test` is fast, deterministic, and model-free; model quality never enters CI.** Real-model work lives behind `b2 init` / the out-of-CI eval. `#[ignore]` is forbidden — a hard-to-write test is a signal to re-anchor on the invariant or fix the system. - ([CLAUDE.md](../CLAUDE.md), [specs/eval-strategy.md](specs/eval-strategy.md)) + ([CLAUDE.md](../../CLAUDE.md), the eval harness under `crates/b2-embed/evals/`) - **E3 — The `Vault` façade is the one typed API, and every adapter is dumb.** CLI and desktop commands are deserialize → one façade call → serialize; logic that wants to live in an adapter belongs behind the façade. Dependencies point one way (adapters → core, never back); façade ops are - added on need, never pre-built. ([crates/b2-desktop/CLAUDE.md](../crates/b2-desktop/CLAUDE.md)) + added on need, never pre-built. ([crates/b2-desktop/CLAUDE.md](../../crates/b2-desktop/CLAUDE.md)) - **E4 — User-facing errors are generic and actionable, never leaking internals.** Full detail goes - to logs / `B2_DEBUG`, not to the terminal or webview. ([CLAUDE.md](../CLAUDE.md) Conventions) + to logs / `B2_DEBUG`, not to the terminal or webview. ([CLAUDE.md](../../CLAUDE.md) Conventions) diff --git a/docs/discovery.html b/docs/discovery.html index ef47a9c..3d23ac9 100644 --- a/docs/discovery.html +++ b/docs/discovery.html @@ -244,7 +244,7 @@

    2What a run costs

- B2 · b2 similar deep dive · a projection of the spec in planning/. Back to the + B2 · b2 similar deep dive · a projection of the spec in docs/design/. Back to the system architecture, the indexing pipeline, or the overview.
diff --git a/docs/index.html b/docs/index.html index 2311403..3417722 100644 --- a/docs/index.html +++ b/docs/index.html @@ -136,11 +136,11 @@

Read the system architecture →

🗂️ Indexing pipeline 🔗 Connection discovery ⌂ Source on GitHub - 📄 Design docs (planning/) + 📄 Design docs (docs/design/)
- B2 · index engine + CLI + desktop app · Rust. The design docs in planning/ are the source + B2 · index engine + CLI + desktop app · Rust. The design docs in docs/design/ are the source of truth; the code is a projection of the spec. Built in the open.
diff --git a/docs/indexing.html b/docs/indexing.html index d70f84a..333410f 100644 --- a/docs/indexing.html +++ b/docs/indexing.html @@ -391,7 +391,7 @@

7What's deferred (and why it's clean)

- B2 · indexing deep dive · a projection of the spec in planning/. Back to the + B2 · indexing deep dive · a projection of the spec in docs/design/. Back to the system architecture or the overview.
diff --git a/fixtures/README.md b/fixtures/README.md index bcbb402..61b1f45 100644 --- a/fixtures/README.md +++ b/fixtures/README.md @@ -7,7 +7,7 @@ Two committed vaults, for two different jobs. This file lives *outside* both — The small, hand-authored vault the **deterministic integration tests** copy into a tempdir and assert against (fixed `b2id`s in `crates/b2-core/tests/common/mod.rs`; shape per -`planning/data-model.md §8`). Model-free — the suite runs the `FakeEmbedder`. Change it only +`docs/design/data-model.md §8`). Model-free — the suite runs the `FakeEmbedder`. Change it only with the tests. ## `test-vault/` @@ -24,7 +24,7 @@ retrieval-quality experiments** that need volume, *not* the deterministic suite: real semantic clusters (vector-search, distributed-systems, rust, pkm, transformers, databases, productivity, gardening, coffee, hiking) cross-linked by body `[[wikilinks]]` (~2,300 edges) and a few typed frontmatter `b2_relations:`. Good for eyeballing `b2 search` / `b2 similar`. It is - **not** the hand-labelled retrieval eval set (`planning/specs/eval-strategy.md`) — a scale + **not** the hand-labelled retrieval eval set (the eval harness under `crates/b2-embed/evals/`) — a scale fixture, not a graded benchmark. The prose is templated (sentences recombined from the pools), not human-authored — realistic diff --git a/planning/research/discovery-scan-strategy.md b/planning/research/discovery-scan-strategy.md deleted file mode 100644 index 953f8fb..0000000 --- a/planning/research/discovery-scan-strategy.md +++ /dev/null @@ -1,200 +0,0 @@ ---- -title: "B2 — Research: making discovery O(fast) — the #38 scan-strategy decision" -type: note -tags: [b2, research, index-engine, sqlite-vec, discovery, performance, quantization, ann] -created: 2026-07-12 -status: draft ---- - -# B2 — Research: making discovery O(fast) — the #38 scan-strategy decision - -> Deep analysis for [#38](https://github.com/AlteredCraft/B2/issues/38): note-open discovery -> (`b2 similar`, the desktop **Similar & Connections** pane) is still ~4.4 s (release) on the -> primary vault after the [#37](https://github.com/AlteredCraft/B2/issues/37) fix. Goal: relation -> resolution for a given document feels **instant**, and stays instant at **10× the current -> document population** (~1k docs → ~10k docs). Constraints: local-first on a laptop, no -> app-managed result cache (OS/library-internal caches are fine), background one-time heavy -> lifts acceptable, maintained derived structures acceptable if low-effort. Full refactor was -> on the table; the conclusion is it isn't needed. - -## TL;DR / recommendation - -**Own the flat scan: store vectors in a plain SQLite table, score them in-process, and drop -`vec0` from the read path entirely. Then make discovery two-stage (note-centroid prefilter → -exact chunk rescore) so the heavy pass is O(notes), not O(chunks).** No new store, no ANN -library, no managed cache. - -1. **Now (fixes #38 and #36 structurally):** replace `chunks_vec` (a `vec0` virtual table) with - a plain `embeddings(chunk_id INTEGER PRIMARY KEY, vector BLOB)` table; scan it with one - sequential statement; fix the arithmetic shape (reused scratch buffer, unrolled accumulators); - add `PRAGMA mmap_size` + a bigger `cache_size` in `db::open`. Expected: **~4.4 s → ~0.2–0.3 s - warm** at today's scale, bit-identical ranking, one SQL log line per open instead of ~38.6K, - and the `sqlite-vec` dependency can be removed outright. -2. **For 10× (small; can ship with 1):** a per-note **centroid** column maintained by the same - embed pass — discovery scans ~N_notes centroids (10k × 768 ≈ 30 MB at 10×, ~5 ms), shortlists - the top ~200 notes, and runs today's exact max-sim only over the shortlist's chunks - (~10–20 ms). Scaling becomes essentially flat in vault size. -3. **Multiplier already on the roadmap:** the qmd chunker upgrade - ([#19](https://github.com/AlteredCraft/B2/issues/19)) shrinks ~38 paragraph-chunks/doc to a - handful of ~900-token chunks — a ~5–40× reduction in vector count that makes every option - cheaper (and max-sim less noisy). Do it for quality; bank the perf. -4. **Held in reserve, in order:** int8/binary **quantization** (shrinks the constant, needs the - eval), **ANN** (usearch/LanceDB — wrong scale, non-deterministic, standing maintenance; - both planning docs already ranked it last), **precomputed similar-notes table** (a managed - cache with an O(N²)-shaped invalidation problem — the explicit last resort). - -## 1. Problem decomposition — which "relations" are slow - -"Resolve the relations of a given document" is two different queries: - -- **Authored relations** (backlinks + outbound, `Vault::neighbors`/`explain`) — indexed lookups - on the materialized `edges` table. Already O(link-degree), microseconds. **Not the problem.** -- **Latent relations** (`b2 similar` / `discover::candidates` — "semantically near ∖ already - connected") — an **exact brute-force max-sim scan of every stored chunk vector on every - note-open**. This is the whole problem, and (post-#37) it is the only remaining O(vault) - read in the open-a-note path. `hybrid_search` and `graph_filtered_search` pay the same - per-row tax on their `vec0` scans, so whatever fixes discovery fixes them for free. - -## 2. What we've learned so far (the #35→#36→#37 trail) - -The issue history localizes the cost precisely; this is worth restating because it *rules out* -most exotic fixes: - -- **#37:** the A× whole-space rescan (one KNN scan per anchor chunk) and the per-hit N+1 were - the first-order bug — fixing the *access pattern* took 51 s → 4.4 s with zero algorithm change. -- **#36 + #38:** what remains is dominated by **how the bytes are read, not how many FLOPs are - done**. Every row read from `chunks_vec` makes `sqlite-vec` probe its `chunks_vec_rowids` - shadow table (~38.6K internal single-row statements per open — the O(vault) log lines), and - the 132 MB DB is walked through SQLite's default ~2 MB page cache with no mmap (the ~3.2 s - that stays *system* time even warm). -- **Arithmetic shape, measured** (microbench at N=38,610 × 768-dim, A=12, this container's CPU, - `rustc -O`): the current shape — per-row `unpack_f32` `Vec` allocation + iterator-`sum()` - `l2_sq` (float non-associativity blocks autovectorization) — costs **~530 ms**; a reused - scratch buffer + 8-accumulator unrolled loop costs **~75 ms** (10×: ~5.2 s vs ~0.74 s). The - index-engine.md §4 prediction ("tens of ms") was about the FLOPs and remains right; the - implementation shape multiplies it ~7×. -- **Structural fact:** `sqlite-vec`'s shipped search is brute force — no ANN path — and - discovery already scores in-process. So on the scan path `vec0` provides **storage only**, - at the price of a per-row vtab round-trip. B2 uses none of what it would charge for. -- **Structural fact #2:** the shipped chunker is the *minimal paragraph splitter* (`chunk.rs`), - not the planned ~900-token qmd heuristic (#19). ~1k docs → 38.6k chunks (~38/doc) means the - vector population is inflated ~5–40× over the design target. #38's cost is partly a chunking - artifact. - -Consequence: **before changing the algorithm, the read path and arithmetic shape are worth -~10–20×.** That reframes the option space — heuristics are for the 10× headroom, not for today's -4.4 s. - -## 3. Budget at target scale - -| | today (~1k docs) | 10× (~10k docs, current chunker) | 10× + #19 chunker | -|---|---|---|---| -| chunks | ~38.6k | ~386k | ~30–80k | -| f32 vectors on disk | ~118 MB | ~1.2 GB | ~90–240 MB | -| flat scan, fixed shape (measured/extrapolated) | ~75 ms + read | ~740 ms + read | ~60–160 ms + read | -| two-stage (centroid → rescore ~200 notes) | ~10 ms | **~15–25 ms** | ~10 ms | - -Brute force with a sane memory layout is comfortably "instant" today and *borderline* at 10× -under the current chunker; the centroid stage (or #19 alone) restores a wide margin. This -matches industry practice: FAISS's own guidance is flat/exact search until high-hundreds-of- -thousands of vectors, and ANN only when exactness or memory must be traded for scale. A -personal vault never leaves the flat regime — *if* the flat scan isn't paying a per-row -virtual-table tax. - -## 4. Options considered - -### A. Own the flat scan (plain table + in-process top-k + pragmas) — **do now** - -Store embeddings in an ordinary table (`embeddings(chunk_id INTEGER PRIMARY KEY, vector BLOB)`, -same packed-LE-f32 blob), keyed/lifecycled exactly as `chunks_vec` is today (created at embed -time at the model's dim; dropped on model swap; `meta` discipline unchanged). All four vector -consumers — `for_each_stored_vector`, `vector_search`, `vector_search_all`, and #36's -missing-vector anti-join — become one sequential B-tree scan (plus, for search, an in-process -top-k heap). Add `PRAGMA mmap_size` (≥ DB size, e.g. 1–2 GB — it's a cap, not an allocation) -and a bigger `page_size`/`cache_size` so the scan streams from the OS page cache — the one -cache we're happy to use because nobody manages it. Fix `l2_sq`/`unpack_f32` shape as measured. - -- **Pros:** simplest possible change; results bit-identical (same vectors, same metric, same - order); kills the O(vault) log lines and #36's probe storm *structurally*; deletes a pre-v1 - dependency (`sqlite-vec`) instead of adding one; keeps the single-store `chunks ⨝ edges ⨝ - embeddings` join property that decided the qmd and Chroma/Lance evaluations; fully - deterministic; no index to maintain beyond what embed already maintains. -- **Cons:** we own ~50 lines of scan/top-k code and the (already-shared) blob format; still - O(chunks) per open — at 10× under the current chunker that's ~0.8–1 s warm, i.e. A alone - doesn't hit the 10× "instant" bar; forfeits a hypothetical future sqlite-vec ANN mode (which - the planning docs already said not to design for). - -### B. Two-stage discovery: note-centroid prefilter → exact chunk rescore — **the 10× lever** - -At embed time, also store one **summary vector per note** (mean of its chunk vectors, -L2-renormalized; summation in `seq` order so it's deterministic). Discovery becomes: (1) scan -~N_notes centroids against the anchor's centroid, excluding the anchor + 1-hop neighbors; -(2) take a generous shortlist (~10–20× the display limit, e.g. 200–300 notes); (3) run today's -exact max-sim over only the shortlist's chunks; surface evidence chunks as now. This is IVF's -coarse-quantizer idea using B2's *natural* partition (the note) instead of learned clusters — -no training, no ANN library, no recall cliff. - -- **Pros:** the heavy pass drops from O(chunks) to O(notes) — ~30 MB / ~5 ms at 10×, flat - effectively forever (100k notes ≈ 300 MB ≈ still sub-100 ms); maintenance is one UPDATE per - note inside the existing embed pass (the same lifecycle as the vectors themselves — a derived - projection in the disposable index, **not** an app-managed result cache: nothing to - invalidate that embed doesn't already touch); deterministic; the note-level vector is also a - natural substrate for #20 (distance weighting) later. -- **Cons:** no longer exhaustively exact — a note whose centroid is far away but which contains - one very close paragraph can miss the shortlist. Mitigations: discovery is *by design* - permissive/recall-oriented with a human precision gate, the shortlist is cheap to widen, and - the retrieval eval (eval-strategy.md) can pin shortlist size against exact-scan ground truth. - Slightly more code than A (two stages instead of one). - -### C. Quantization (int8 / binary + f32 rescore) — **reserve** - -`index-engine.md` §4 names it the first lever before ANN. Once we own the scan (A), an extra -int8 (4×) or binary (32×, Hamming/popcount, rescore top-k in f32) column is easy to add and is -standard practice for bge-family embeddings. - -- **Pros:** shrinks bytes-read and FLOPs by 4–32× (1.2 GB → 300 MB / 37 MB at 10×); composes - with A and B; well-trodden (FAISS SQ8, binary-quantized bge retains ~95 %+ recall with - rescoring). -- **Cons:** measurable quality risk → gated on the eval harness; more code than B for a smaller - win than B at this scale; reduces the constant, not the O(chunks) growth. Only worth it if, - after A+B(+#19), cold-start I/O or memory footprint still bites. - -### D. Real ANN (HNSW via usearch/hnsw_rs, IVF, or LanceDB vector tier) — **rejected** - -- **Pros:** sub-linear queries; the industry default at 10M+ vectors. -- **Cons:** solves a scale B2 doesn't reach at 10× (§3); HNSW graphs are build-order-dependent - (breaks the determinism requirement); a second index artifact to build/persist/heal alongside - the disposable-projection invariant; dependency weight vs. the single binary; both - `index-engine.md` §4 and `research/vector-store-alternatives.md` already ranked ANN last and - re-affirmed staying on SQLite. Nothing in #38's evidence (read-path overhead, not FLOPs) - argues for it. - -### E. Materialized similar-notes table (precompute discovery per note) — **last resort, as specified** - -Background-compute top-k candidates per note at embed time; note-open reads a row. - -- **Pros:** O(1) note-open, trivially instant. -- **Cons:** this is exactly the app-managed cache the requirements deprioritize: one note's new - embedding can perturb *every* other note's candidate list, so correctness needs either O(N²) - recompute sweeps or push-based incremental maintenance — a standing effort/complexity bill. - B delivers ~tens-of-ms opens without any of that; E only becomes interesting if the target - were single-digit ms on a vastly larger corpus. - -## 5. Sequencing & migration - -1. **A** — new `embeddings` table + in-process top-k + pragmas + arithmetic shape; delete - `vec0`/`sqlite-vec`. Bump `schema_version`: the existing migration gate drops derived tables - and the next `reindex` rebuilds — and thanks to the projection/embedding split the app stays - usable (BM25 + graph) while the background embed refills vectors. That *is* the acceptable - "one-time heavy lift, slightly diminished state". (If re-embedding ~38.6k chunks is - undesirable, a one-shot copy-across from `chunks_vec` before dropping it avoids even that.) - Closes #38's acceptance criteria at today's scale and #36 as a side effect. -2. **B** — centroid column + two-stage `discover::candidates`, shortlist size pinned by the - eval against exact ground truth. Buys the 10× (and 100×) headroom. -3. **#19** — the chunker upgrade, on its own quality-driven schedule; it multiplies both wins. -4. **C** then **D** stay shelved until the eval + measured latency say otherwise; **E** stays - the last resort. - -Regression guardrails, matching the #37 pattern: extend `discover_query_count.rs` to assert the -whole-space pass emits O(1) SQL statements (not O(chunks)), and add a scale smoke-test with a -synthetic few-hundred-note vault under the fake embedder. diff --git a/planning/research/file-type-support.md b/planning/research/file-type-support.md deleted file mode 100644 index 919bfea..0000000 --- a/planning/research/file-type-support.md +++ /dev/null @@ -1,505 +0,0 @@ ---- -b2id: 01KX1TDVWHCXBM4XCN1GEWBXAB -title: "B2 — Beyond Markdown: file-type support (resources)" -type: note -tags: [b2, resources, file-types, images, pdf, html, rendering, ingestion, research] -created: 2026-07-08 -status: draft ---- - -# B2 — Beyond Markdown: file-type support (resources) - -> **Findings + design for supporting non-`.md` files in the vault.** Centralizes the analysis -> scattered across the shipped specs and issues (§1), derives the model from the two tenets -> ([vision-and-scope.md](../vision-and-scope.md#design-philosophy)), and specifies one polymorphic -> approach covering ingestion (§3–§5), the graph (§4), and rendering with a "no viewer available" -> fallback (§6). The bar: **any file type GitHub could store** — the taxonomy must be *total*, with -> graceful degradation, never a refusal. The §9 judgment calls were **resolved 2026-07-08**; the -> design below is written in its locked form. - -## Rollout — propagation & build (todo) - -The design is **locked** (§9); what remains is propagating it into the canonical docs, then the code -slices (§8). Tracked here so state survives across sessions: - -- [x] **Stage A — mirror the locked model into the canonical docs** (2026-07-12) — the `resource` - object + widened invariant → [data-model.md](../data-model.md) §10; the `resources` table + the - per-class extraction step → [index-engine.md](../index-engine.md) §3; the locked decisions → - [vision-and-scope.md](../vision-and-scope.md) "Decisions locked (2026-07-08)"; the working-queue - pointer → [tasks.md](../tasks.md). -- [x] **Reconcile with [#38](https://github.com/AlteredCraft/B2/issues/38)** (2026-07-12) — the design - predates the vector-store rewrite that landed 2026-07-12 (`sqlite-vec` dropped; plain - `embeddings` + `note_centroids` tables scored in-process; **two-stage discovery** — centroid - shortlist → exact rescore; [discovery-scan-strategy.md](discovery-scan-strategy.md)). Absorbed - here: the multimodal seam is a second *plain* vector table set, not a `vec0` table (§5); and - **centroids generalize alongside chunks** — discovery's coarse stage scans only centroids, so - `b2 similar` over resources requires document-keyed centroids, not just document-keyed chunks - (§5). No locked decision changes. -- [x] **Slice-prep decisions locked** (2026-07-12, §9b #7–#10) — chunk/centroid DDL (one-of FKs + - sibling `resource_centroids`), anchor dispatch by extension, direction-agnostic 1-hop - exclusion, `Vault::list_resources`. -- [x] **Stage B — slice-1 build spec** (2026-07-12) — - [specs/resources-inventory-graph.md](../specs/resources-inventory-graph.md): inventory & graph - (§8 slice 1) — the v4 schema (`resources` + `edges` widening), the generalized walk with - pruning + hashing, the parser forms with caption capture, the façade additions - (`list_resources` / `explain_resource` / `move_resource` / `doc_kind`), CLI dispatch, desktop - tree + fallback card + watcher inversion. **Model-free; no new engine deps** (the desktop - gains the Tauri opener plugin — the one new adapter dep). Next: build it. -- [x] **Slice 1 built** (2026-07-12) — steps 0–7 of the spec, whole-workspace green + CLI smoke; - pending a live desktop dogfood before the spec moves to `completed/`. -- [ ] **Slices 2–4** — render mechanisms · searchable resources · PDF text (§8); spec each when reached. -- [ ] **Slice 5** — semantic seams (Describer, multimodal embedder), future/unscheduled (§8, §5). - -## TL;DR / recommendation - -**Markdown is the vault's only *authoring surface*; every other file is a *resource* — a -first-class vault member in its own right: indexed, searchable, linkable, renderable, and never -required to be referenced by any note.** - -- **The two-tier model is unchanged; the source tier widens.** The invariant generalizes from - `index = projection of (the .md files)` to **`index = projection of (the vault directory)`** — - resources contribute only *derived* index rows (metadata, extracted text, inbound edges), never - durable state. Drop `b2.sqlite`, reindex, get it back identical. No new tier, no sidecar files. -- **Polymorphism = a closed class table with a total fallback**, not a trait hierarchy. Every file - maps by extension to one of six classes — `note`, `text`, `html`, `pdf`, `image`, `media` — with - **`binary`** as the catch-all. Each class answers the same three questions: *what text does it - yield for the index?* (§5), *can it be a graph endpoint?* (§4), *how does it render?* (§6). -- **Resources are path-keyed peers with one asymmetry: authoring.** No `b2id` (nothing to stamp a - PNG with) and no outbound edges in v1 — not because resources are subordinate, but because B2 - can only read and write *authored structure* in Markdown, and a PNG has no home for it. A - resource needs no note to exist, be indexed, be found, or be opened. Where edges touch a - resource they originate in notes for now (`![[photo.png]]`, `[[papers/x.pdf]]`, `b2 link`), - with named relief valves if resource-sourced edges ever prove needed (§4). -- **One embedding space in v1.** Every class funnels to *text* (native, extracted, or — for images — - aggregated alt-text/captions from the notes that embed them), embedded in the existing bge space. - Multimodal image embedding is a **documented future seam** (a second plain vector table set under - the same `meta` discipline, #38-style), not a v1 build — the Bitter-Lesson posture that cut the - relator (§7). -- **Rendering = a viewer registry keyed by class, with a fallback card.** Selecting any file in the - tree opens *something*: the note pane for `.md`, an ``/`