Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ crates/b2-desktop/gen/
# don't ignore logs/.keep
!/logs/.keep

# The retrieval eval's append-only results log (specs/eval-strategy.md): one JSON
# The retrieval eval's append-only results log (the eval harness, crates/b2-embed/evals/): one JSON
# line per scored run, machine-local (scores depend on the machine's model/device).
crates/b2-embed/evals/results.jsonl

Expand Down
35 changes: 16 additions & 19 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,26 +11,23 @@ A personal, local-first Markdown knowledge vault with an AI layer that **surface
notes** for you to connect. The Markdown files stay plain and yours; B2 is the intelligence layer
over them, not a container around them. This Cargo workspace is the **index engine + its two dumb
adapters** — the `b2` CLI and the Tauri desktop app (with the `ui/` frontend); the design lives in
`planning/`.
`docs/design/`.

## Design docs are the source of truth

The code is a *projection of the spec*, and comments cite it constantly (e.g. `data-model.md §2`,
`build spec §1.2`, `index-engine.md §6`). Before changing behavior, read the relevant doc — the
schema must satisfy the data model, never the reverse.

- `planning/invariants.md` — the **invariant register**: the one-page normative list of what must
always be true (cited by id — S2, G2, …). On conflict with any other doc, it wins.
- `planning/vision-and-scope.md` — the *why*: principles, the two design tenets, v1 scope, locked decisions.
- `planning/data-model.md` — the *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary.
- `planning/index-engine.md` + `planning/specs/completed/index-engine-build.md` — the *how*: SQLite (FTS5 + in-process vector scan; see `research/discovery-scan-strategy.md`) projection, table DDL, the build order, data flows.
- `planning/tasks.md` — the working queue (what's done, what's next). **Read this first to know current state.**
Planned-but-unstarted work lives in GitHub Issues; shipped build specs live in `planning/specs/completed/`
(index engine, desktop MVP, async indexing, projection/embedding split, desktop editing, live preview).
- `planning/user-stories.md` — kernel behavior as testable scenarios.
- `planning/specs/eval-strategy.md` — how model quality (the `Embedder` seam) is measured out-of-CI:
the hand-labelled retrieval + discovery evals (BM25-vs-hybrid ablation, note & passage ranks,
`b2 similar`), the chunker-sweep gate, the results log, and how to run/grow it all.
`index-engine.md §6`). Before changing behavior, read the relevant doc — the schema must satisfy the
data model, never the reverse. The three canonical docs live in `docs/design/`:

- `docs/design/invariants.md` — the **invariant register**: the one-page normative list of what must
always be true, and the source of *why* (cited by id — S2, G2, …). On conflict with any other doc, it wins.
- `docs/design/data-model.md` — the *what*: note + connection in Markdown, the two storage tiers, the relation vocabulary.
- `docs/design/index-engine.md` — the *how*: SQLite (FTS5 + in-process vector scan) projection, table DDL, data flows.

Planned-but-unstarted work and the backlog live in [GitHub Issues](https://github.com/AlteredCraft/B2/issues);
shipped build history lives in git. Model quality (the `Embedder` seam) is measured out-of-CI by the
eval harness under `crates/b2-embed/evals/` — the hand-labelled retrieval + discovery evals
(BM25-vs-hybrid ablation, note & passage ranks, `b2 similar`), the chunker-sweep gate, and the results log.

## Commands

Expand Down Expand Up @@ -110,7 +107,7 @@ so Tauri/wry tracing doesn't pollute the file (an explicit `B2_LOG` is honored v

### The core invariant

**`index = a pure projection of (the vault directory)`.** (The full register: `planning/invariants.md`.) Two storage tiers:
**`index = a pure projection of (the vault directory)`.** (The full register: `docs/design/invariants.md`.) Two storage tiers:

1. **The vault directory** — the source of truth. **Markdown is its sole authored subset** — the only
format whose bytes B2 may write; non-`.md` files are *resources* (path-keyed peers contributing
Expand Down Expand Up @@ -187,7 +184,7 @@ adapters wire the real model.
- **Flow ① ingest/reindex** (`ingest.rs`) — parse → stamp missing `b2id` (write file) → project
notes, chunks (+FTS), embeddings, and the typed `edges` graph. Two-phase so link resolution is
independent of file order. It is **two separately-invokable passes**
(`specs/completed/projection-embedding-split.md`): model-free `project_vault` (notes/chunks/FTS/edges) and
(the `project`/`embed` split, #15): model-free `project_vault` (notes/chunks/FTS/edges) and
`embed_vault` (fills the DB-derived missing-vector set); `reindex` composes them, and `search`
falls back to BM25-only on a projected-but-unembedded vault.
- **Flow ② hybrid search** (`search.rs`) — BM25 (`chunks_fts`) ⊕ vector KNN (an exact in-process scan
Expand Down Expand Up @@ -225,7 +222,7 @@ Vectors live in **plain tables** — `embeddings(chunk_id, vector)` and `note_ce
centroid)` — created at **embed time**, not in the base migration: their existence is the "this vault
has an embedding space" signal the projected-but-unembedded fallbacks key on. Every distance is
computed **in-process** (`embed::l2_sq`, one sequential scan statement; rationale:
`research/discovery-scan-strategy.md`, #38). `meta` records `(embed_model_id,
#38). `meta` records `(embed_model_id,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Avoid a bare #38 at the start of the Markdown line.

markdownlint reports MD018 because this line is parsed as an ATX heading without a space after #. Keep the issue reference inline or rewrite it as GitHub issue 38.

🧰 Tools
🪛 markdownlint-cli2 (0.23.0)

[warning] 225-225: No space after hash on atx style heading

(MD018, no-missing-space-atx)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CLAUDE.md` at line 225, Update the Markdown text around the issue reference
so it does not begin with the bare “#38” sequence; keep the reference inline
with surrounding text or rewrite it as “GitHub issue 38” while preserving the
original meaning.

Source: Linters/SAST tools

embed_dim)` — the only place a model swap is detectable. The compute **device** folds into this
identity: the real embedder tags its recorded `embed_model_id` with the resolved device (CPU stays the
bare repo id; a `--features metal` GPU build appends `@metal`, `b2-embed/src/model.rs`), so a
Expand Down
8 changes: 4 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,17 @@ resolver = "2"
members = ["crates/b2-core", "crates/b2-embed", "crates/b2-cli", "crates/b2-desktop"]

# B2 — local-first Markdown PKM with AI connection discovery.
# Design lives in planning/; this workspace is the index engine
# (planning/specs/index-engine-build.md), built step 0→5 against the
# golden-vault fixtures in fixtures/golden-vault/ (planning/data-model.md §8).
# Design lives in docs/design/; this workspace is the index engine
# (docs/design/index-engine.md), built step 0→5 against the
# golden-vault fixtures in fixtures/golden-vault/ (docs/design/data-model.md §8).

# Build DEPENDENCIES optimized even in dev/debug builds, while keeping our own `b2-*`
# crates at opt-level 0 so the TDD loop and `cargo test -p b2-core` stay fast. Candle's
# embedder forward pass (matmul + transformer ops, spread across candle *and* its gemm
# matmul backend) is ~13× slower unoptimized, so a plain `tauri dev` / `cargo run`
# reindex was painfully slow (a 16-chunk embed batch took ~35s instead of ~2.5s). That
# mattered doubly for the desktop app: a reindex cancel is only observed at each embed-
# batch boundary (planning/specs/async-indexing.md §3), so a slow batch made the
# batch boundary (docs/design/index-engine.md), so a slow batch made the
# **Cancel** button feel stuck. The `"*"` glob optimizes every dependency (targeting
# just the candle crates missed the matmul backend and left it ~4× slower); the per-
# crate opt-0 overrides below exclude our own crates so their rebuilds stay instant.
Expand Down
25 changes: 11 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ A personal, **local-first** knowledge vault — plain Markdown you fully own —
explained connections between them yourself.

> **Status:** the design is **locked** and the **index engine is built** (`crates/b2-core`: steps 0→5
> of the [build spec](planning/specs/completed/index-engine-build.md)). The **`b2` CLI over a typed core API** is
> of the index engine). The **`b2` CLI over a typed core API** is
> live (`crates/b2-cli`): point B2 at a folder and `reindex` / `search` / `neighbors` / `explain` it
> from the terminal, with `--json` for agents. **Semantic search is real** (`crates/b2-embed`: a
> candle-backed local embedder behind the one seam; `b2 init` downloads the model into a shared cache;
Expand All @@ -31,10 +31,10 @@ explained connections between them yourself.
> reconciles external edits live. Reindex is a **cancellable background action** — live progress, a Cancel
> button, and the UI stays usable while a large vault indexes; projection and embedding are decoupled, so
> a cold vault is browsable/keyword-searchable in seconds while embedding streams behind
> ([specs/completed/](planning/specs/completed/)). Run it with `just app` — pick a vault from the
> ([#15](https://github.com/AlteredCraft/B2/issues/15)). Run it with `just app` — pick a vault from the
> in-app switcher, or skip straight to one via `B2_VAULT_PATH`. **Next:** file-type support (resources) —
> slice 1, inventory & graph, is built; the wider backlog lives in
> [GitHub Issues](https://github.com/AlteredCraft/B2/issues) ([tasks.md](planning/tasks.md)).
> [GitHub Issues](https://github.com/AlteredCraft/B2/issues).

## What B2 is (the north star)

Expand All @@ -44,12 +44,12 @@ connected yet** — so the structure of your knowledge grows as you link them, i
The files stay plain Markdown on your disk, yours forever; B2 is the **intelligence layer over them,
not a container around them**. Humans and AI agents are both first-class users.

Full motivation, scope, and locked decisions: **[vision-and-scope.md](planning/vision-and-scope.md)**.
Full motivation, scope, and locked decisions: **[docs/design/invariants.md](docs/design/invariants.md)**.

## How we build it

Two architectural tenets shape every decision (full text:
[vision-and-scope.md → Design philosophy](planning/vision-and-scope.md#design-philosophy)):
[docs/design/invariants.md](docs/design/invariants.md)):

- **A volatile vault over a disposable index.** Refactor fearlessly — move, split, merge, compress,
trim orphans. The index is a pure projection of your vault (drop it, rebuild it identical);
Expand All @@ -61,7 +61,7 @@ Two architectural tenets shape every decision (full text:

…in service of five product non-negotiables — plain-Markdown source of truth · local-first · zero
lock-in · AI-native (not bolted-on) · single binary
([vision-and-scope.md → Principles](planning/vision-and-scope.md#principles--non-negotiables)).
([docs/design/invariants.md](docs/design/invariants.md)).

## The docs

Expand All @@ -75,14 +75,11 @@ and work with a vault in about ten minutes. Then go deeper:

| Doc | What it owns |
|---|---|
| [invariants.md](planning/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, cited by id. On conflict with any other doc, it wins. |
| [vision-and-scope.md](planning/vision-and-scope.md) | Why B2 exists · principles · **design philosophy** · v1 scope · locked decisions. The canonical *why*. |
| [data-model.md](planning/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. |
| [index-engine.md](planning/index-engine.md) | How the derived index is *built* — SQLite (FTS5 + an in-process vector scan) as a disposable projection. The canonical *how*. |
| [specs/index-engine-build.md](planning/specs/completed/index-engine-build.md) | The build **spec** — precise table DDL, relations, data flows, and the step 0→5 build order. The buildable contract. |
| [specs/completed/desktop-ui-mvp.md](planning/specs/completed/desktop-ui-mvp.md) | The **desktop UI** build spec — Tauri + CodeMirror, the repo layout, the thin-adapter discipline, and the read→discover→link MVP. The first UI adapter. |
| [user-stories.md](planning/user-stories.md) | Kernel behavior as testable scenarios (rename/move, link delete) · link-identity mechanics. |
| [tasks.md](planning/tasks.md) | The working queue — what's done, what's next. |
| [docs/design/invariants.md](docs/design/invariants.md) | The **invariant register** — the one-page normative list of what must always be true, and the source of *why*, cited by id. On conflict with any other doc, it wins. |
| [docs/design/data-model.md](docs/design/data-model.md) | What a **note** and a **connection** are, in plain Markdown · the two storage tiers · the relation vocabulary · the invariant *definitions*. The canonical *what*. |
| [docs/design/index-engine.md](docs/design/index-engine.md) | How the derived index is *built* — SQLite (FTS5 + an in-process vector scan) as a disposable projection. The canonical *how*. |

Planned work and the backlog live in [GitHub Issues](https://github.com/AlteredCraft/B2/issues); shipped build history lives in git.


## Build and run
Expand Down
12 changes: 6 additions & 6 deletions crates/b2-cli/src/main.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//! `b2` — the first adapter over the `b2-core` typed API (vision-and-scope,
//! `b2` — the first adapter over the `b2-core` typed API (invariants.md,
//! headless-first: "the CLI is the UI before the UI"). It holds **no engine logic**:
//! it parses args, picks + injects the embedder, calls the [`Vault`] façade, and
//! prints — human-readable by default, or `--json` for agents.
Expand All @@ -25,7 +25,7 @@ use std::sync::atomic::{AtomicBool, Ordering};
/// Set by the Ctrl-C handler installed for a foreground `reindex`. The embed loop
/// reads it at each batch boundary (through the [`ControlFlow`] the progress closure
/// returns — the shipped cancel seam) and stops *after* the current batch: a
/// consistent, re-runnable partial index, never a torn write (async-indexing.md §3/§8).
/// consistent, re-runnable partial index, never a torn write (index-engine.md).
static CANCEL: AtomicBool = AtomicBool::new(false);

/// Map the Ctrl-C flag onto the reindex embed loop's cooperative-cancel signal —
Expand Down Expand Up @@ -336,7 +336,7 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> {
if p.note_chunks == 1 { "" } else { "s" },
);
let _ = std::io::stderr().flush();
// Stop after this batch if Ctrl-C was pressed (async-indexing.md §3/§8),
// Stop after this batch if Ctrl-C was pressed,
// else carry on. The batch is already written above, so a cancel here
// never tears a write.
cancel_flow()
Expand Down Expand Up @@ -388,8 +388,8 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> {
Command::Status => {
// Read-only coverage report: how much of the vault is embedded (semantic
// ranking live vs. keyword-only) and whether a background reindex is in
// flight — the companion to backgrounding a slow reindex with `b2 reindex &`
// (async-indexing.md §8). A pure model-free DB read (#26): open with the fake.
// flight — the companion to backgrounding a slow reindex with `b2 reindex &`.
// A pure model-free DB read (#26): open with the fake.
let root = cli.vault_or_cwd();
let (vault, _semantic) = open_vault(&root, false)?;
let status = vault.embed_status()?;
Expand Down Expand Up @@ -543,7 +543,7 @@ fn dispatch(cli: &Cli) -> Result<(), CliError> {
}
}
// If nothing points *at* the note, it's an orphan — surfaced, not
// acted on (user-stories.md Story 2; files are only touched when asked).
// acted on (invariants.md; files are only touched when asked).
if !view.connections.iter().any(|c| c.direction == "inbound") {
println!("No inbound links — this note is an orphan.");
}
Expand Down
4 changes: 2 additions & 2 deletions crates/b2-cli/tests/cli.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
//! CLI-level tests: run the built `b2` binary against a temp copy of the
//! golden-vault fixture and assert its output — the "run a command against a
//! fixture, assert the output" surface vision-and-scope names. The binary path is
//! fixture, assert the output" surface invariants.md names. The binary path is
//! `CARGO_BIN_EXE_b2`, which cargo provides to integration tests (so no extra test
//! harness dependency is needed). The CLI is a dumb adapter over `b2_core::Vault`;
//! these prove the wiring + output shape, not engine behavior (that's the façade
Expand Down Expand Up @@ -38,7 +38,7 @@ fn copy_dir(src: &Path, dst: &Path) {

/// Run `b2 <args...>` and capture the result. The suite runs under the fake
/// embedder (`B2_EMBEDDER=fake`) so CI never downloads or runs the real model — it
/// proves the wiring + output shape, not model quality (tasks.md testability 4–5).
/// proves the wiring + output shape, not model quality (CLAUDE.md).
fn run(args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_b2"))
.env("B2_EMBEDDER", "fake")
Expand Down
2 changes: 1 addition & 1 deletion crates/b2-core/src/add.rs
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ pub fn add_note(
/// [`ingest::project_file`] (chunks + FTS + edges, **no embedder**), the same pass
/// `Vault::write` runs after a save. The new note's chunks join the DB-derived
/// missing-vector set for any later embed/reindex to fill
/// (projection-embedding-split.md §7.2) — and a body-less note has nothing to
/// (index-engine.md) — and a body-less note has nothing to
/// embed anyway. Same validation and refusals as [`add_note`].
#[allow(clippy::too_many_arguments)]
pub fn create_note(
Expand Down
3 changes: 1 addition & 2 deletions crates/b2-core/src/chunk.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
//! Body chunker — the qmd heuristic (index-engine.md §1; spec
//! planning/specs/completed/qmd-chunker.md, issue #19).
//! Body chunker — the qmd heuristic (index-engine.md §1, issue #19).
//!
//! Splits a note body into **size-targeted, overlapping, Markdown-aware** chunks
//! that each carry a `heading_path` breadcrumb (the H1 › H2 › H3 stack the chunk
Expand Down
Loading