Memory Recall

Memory Recall (BM25 over the curated corpus)

The SessionStart snapshot pre-loads everything an agent needs to start work. But once a project's corpus crosses ~50 documents, there is a second question: "Where did we decide X?" or "What do we know about Y?" The agent cannot rely on the snapshot alone; the answer is in the corpus, just not in the always-loaded tier. Memory recall is the second-tier retrieval layer for exactly this case.

dreamcontext memory recall "how did we decide on the sleep fan-out"

Top-5 hits across knowledge files, feature PRDs, task files, 2.memory.md sections (Decisions + Known Issues), and CHANGELOG.json entries. Snippets included. Under 100ms on a 40-doc corpus. No init step, no daemon, no API key.

Why BM25 and not a vector store / mem0

The original exploration was a full mem0 integration: Python + Ollama runtime, LLM-extracted facts on every add(), vector store for semantic recall. Three independent adversarial reviewers (critic, pragmatist, security) converged on rejecting it. The decisive argument:

dreamcontext's content is already curated atomic facts. Knowledge docs are written deliberately. PRDs follow a structured template. Closed tasks have changelogs. The 2.memory.md file is LIFO-ordered, hand-vetted decisions. The LLM extraction step that mem0 provides is solving a problem dreamcontext has already solved at write time. Stacking a non-deterministic 1.5–4s LLM call on top of already-curated content is paying twice for the same value.

Specific issues the reviewers flagged with the vector approach:

• Runtime cliff. Python + Ollama dependency for what is otherwise a Node CLI. Cold start, model download, port management, all for a 44-doc corpus.
• Non-determinism. Same query, different ranking depending on which facts mem0 happened to extract from your input. Hard to debug, hard to test, hard to trust.
• Cost. Every add() is an LLM call. Every recall is a vector search. For content that does not need semantic generalization (most slugs and decisions are exact-match queries), the spend buys little.
• Security surface. Embedding inversion, redaction order bugs, finalizer crash on rebase, and OpenAI exfil paths if a cloud embedding provider was used. Five critical hardening items just to make it safe.
• Documented dedup unreliability. mem0's own native dedup is documented as best-effort. Curated knowledge corpora need reliable dedup, not best-effort.

BM25 over the curated corpus gives ~80% of the value at 1% of the complexity: zero new npm dependencies, deterministic ranking, version-controllable, instant. The full decision trace lives at _dream_context/core/features/memory-recall-bm25.md.

Hybrid recall — the local embedding overlay (experimental, opt-in)

The mem0 rejection named its own escape hatch: an overlay on the BM25 layer, not a replacement, pure-Node, local. That overlay now exists — off by default, enabled with dreamcontext recall hybrid (or DREAMCONTEXT_RECALL_MODE=hybrid).

What it is: multilingual-e5-small (384-dim, ~113 MB quantized ONNX) running in-process via @huggingface/transformers — no Python, no daemon, no API key; vectors never leave the machine. Docs are chunked at markdown heading boundaries (~200–512 tokens, code-fence-aware) and cached by content hash in _dream_context/.embeddings/ (self-gitignoring — vectors are partially invertible, so the cache is treated as credential-class). Refresh is incremental: an edit re-embeds only its changed chunks — lazily at every hybrid query (mtime+size pre-filter, ~15 ms when nothing changed), eagerly with a full re-check at sleep done, and manually via dreamcontext embed refresh [--force].

Fusion is confidence-adaptive, and every piece was benchmark-driven: plain RRF (the textbook choice) was measured first and killed — it regressed exact-term recall@1 from 100% to 83% because rank fusion erases BM25's score margins. What shipped instead: when BM25's top raw score is decisive, score-preserving relative fusion (dense cannot flip an exact-token win — those queries stay byte-identical to BM25); when BM25 is weak, weighted rank fusion lets dense rescue buried docs, with a pin guard protecting internally-confident BM25 top-1s. Dense/RRF feeds rankScore only — the raw BM25 score the hook gates on is untouched in every mode.

Measured on the frozen 60-query train + 30-query held-out gold sets (tuned on train only): overall recall@1 +5.0/+6.7 pts, held-out recall@5 90→96.7, Turkish recall@1 doubled (20→40, r@5 90→100 — the multilingual space bridges TR→EN natively, no LLM call), English paraphrase recall@1 +16.7, and not one recall@k or MRR cell regressed on either set. Dense-only was also measured and is far worse than BM25 alone (38% r@1) — which is exactly why this is an overlay, never a replacement. If the model isn't installed (@huggingface/transformers is an optionalDependency), hybrid mode silently falls back to plain BM25. Full numbers: eval/RESULTS.md ("Embedding A/B").

Note: hybrid replaces the per-prompt Haiku call — recall becomes fully local and deterministic. What you give up is Haiku's judgment (its ability to answer "this prompt has no searchable intent" with silence); what you gain is zero token cost and native cross-lingual matching.

Second consumer: the semantic dedup gate. The index is no longer recall-only. Before a sleep sub-agent creates a knowledge/feature doc, dreamcontext embed dedup scores the candidate against the existing corpus — replacing a keyword-guessing gate that could never work, because you cannot recall a doc you didn't think to search for. The candidate isn't in the corpus, so denseRank can't be reused: it's chunked like an indexed doc, embedded as a passage (same E5 space, both sides passage:-prefixed), and each existing doc scored as the max cosine over all (candidate-chunk × doc-chunk) pairs — best-passage matching on both sides, so a candidate whose one section duplicates one section of an existing doc is caught even when the rest differs. Verdicts: MERGE (near-verbatim twin, above threshold and a top1−top2 margin), REVIEW (same-topic band), CREATE. The module advises only — the fold-in is the agent's knowledge merge/Edit; a specialist's write is never silently rewritten. --if-present no-ops when the vault has no cache or the model isn't installed, so sleep never triggers a first-time model download and falls back to keyword recall.

What is indexed

The corpus is deliberately narrower than "everything in _dream_context/":

Type	Source	Doc unit
knowledge	_dream_context/knowledge/*.md	1 doc per file
feature	_dream_context/core/features/*.md	1 doc per file
task	_dream_context/state/*.md	1 doc per file
memory	_dream_context/core/2.memory.md	1 doc per H2 section (Decisions, Known Issues — LIFO section removed 2026-05-23)
changelog	_dream_context/core/CHANGELOG.json	1 doc per entry; body = summary + description + references[] joined

0.soul.md, the per-person constitutions in people/, the remaining core 3–6 files, RELEASES.json, and the sleep state are intentionally not indexed. They are always-loaded in the snapshot and belong to the deterministic tier. Recall is a complement to the snapshot, not a replacement. CHANGELOG joined the corpus 2026-05-23 because (a) the snapshot's tiered changelog block only surfaces the top 13 entries, so older history needs an on-demand path, and (b) memory remember now writes CHANGELOG entries directly, making CHANGELOG the durable home for quick-capture notes.

CHANGELOG schema (2026-05-23 additions, all optional):

Field	Type	Purpose
summary	string (≤200 char soft cap)	One-line headline rendered in the snapshot's tiered display and in recall snippets.
references[]	string[]	Prefixed evidence tokens: commit:<sha>, file:<path>, knowledge:<slug>, feature:<slug>, task:<slug>, url:<href>. Searchable.
supersedes	string (entry id)	Points at a prior entry this decision replaces — surfaces "this was overridden" relationships during recall.

Ranking

Recall runs on two deliberately decoupled scores, and keeping them separate is the load-bearing design choice of the v0.6.0 engine overhaul:

• score — the raw flat-haystack BM25. Standard BM25 (k1=1.5, b=0.75) over the unweighted union of (title + description + tags + body), with IDF in the log(1 + (N - df + 0.5) / (df + 0.5)) form so it stays non-negative on small corpora. This is the number the hooks gate on (memory-recall injection at ≥ 2.0, the skill gate at ≥ 1.0). Field weighting, recency, and synonyms never touch it — so those thresholds mean the same thing from one release to the next.
• rankScore — the derived sorting signal. This is what hits are actually ordered by. It layers BM25F field weighting (title ×3, tags ×2, description ×2, body ×1 — short, high-signal fields win) on top of the base, then folds in recency (a gentle tie-breaker with a floor, never a content override), task/status relevance, query-time synonym matches, an exact-identity boost for slug/title hits, and [[wiki-link]] connectivity. Auto-captured digests carry a CAPTURE_RANK_PENALTY so hand-curated docs outrank machine-captured ones — again, only here, never in the threshold-bearing score.

The tokenizer is light and bilingual: standard English stopwords plus Turkish particles (ve, ile, ki, için, gibi), since the user codes in mixed Turkish/English. Conservative EN+TR stemming is applied to both the index and the query — it only collapses inflections to a shared stem (databases → database), so identical text still scores identically and the hard score thresholds are unaffected; slug-like terms keep their exact-identity boost on top. A small, hand-curated synonym table (recall-synonyms.ts) bridges common short forms to the corpus's canonical vocabulary, expanding query terms into rankScore only. There is still no embedding model and no semantic recall by design: a curated synonym list is deterministic and inspectable — "ML practitioner" matches "data scientist" only if you put it in the table. If that gap ever proves painful, a deterministic local-embedding overlay (@xenova/transformers with a 30 MB MiniLM model, no Python dep) is the v2 path — not mem0.

Snippet extraction picks the line with the most query-term hits and includes ±1 line of context.

There is no persistent index file. The inverted index is rebuilt in memory on every recall call. For corpora up to ~500 docs the rebuild is under 100ms; storing an index file would introduce gitignore complications and cache-invalidation bugs for negligible speedup.

Command surface

Command	Purpose
memory recall <query...> [--top N] [--types ...] [--json] [--plain]	BM25 search; --top clamped to 1–50, default 5
memory remember <text...> [--type T] [--scope S] [--summary] [--references] [--supersedes]	Write a CHANGELOG entry. Defaults: type=note, scope=quick. Replaces the old LIFO append to 2.memory.md (LIFO section removed 2026-05-23).
memory update <slug> [--description] [--tags] [--content] [--append] [--pin|--unpin]	Edit an indexed doc by slug
memory delete <slug> [--force]	Remove an indexed doc
memory list [--types ...]	List indexed docs (no scoring)
memory status	Corpus stats broken down by type

--json returns { score, type, path, title, snippet } per hit for scripted use. --plain strips ANSI for piping into other tools or grep. --types accepts a comma-separated subset of knowledge,feature,task,memory,changelog.

Hook integration (default ON)

The UserPromptSubmit hook handler injects the top-3 recall hits into the agent's context for every non-trivial user prompt. ON by default; opt out with DREAMCONTEXT_MEMORY_HOOK=0. Originally shipped opt-in per the security reviewer's recommendation, but flipped to default-on the same day (2026-05-23) after the noise/utility tradeoff proved favourable on the live dreamcontext benchmark — strong hits (score ≥2.0) consistently surfaced the right docs, and short prompts are filtered.

Behavior:

• Reads the prompt from stdin (the Claude Code hook payload).
• Skips if the prompt is shorter than 8 characters (filters "hi", "ok", short replies).
• Runs bm25Search(prompt, corpus, 3) against the same corpus as memory recall.
• Emits the block only if the top hit's score ≥ 2.0 (filters weak matches that would just be noise).
• Output is a 5–9 line context block prefixed with — Memory recall (BM25, top 3) —.
• Wrapped in try/catch — recall is best-effort and never breaks the user-prompt flow.

Verified latency budget is under 100ms on a 44-doc corpus with no persistent index.

Why this fits the broader architecture

Memory recall mirrors the pattern the rest of dreamcontext uses: deterministic structure beats clever retrieval, and humans plus agents both need to read what is happening. The corpus is files in your repo. The ranker is the standard formula every information-retrieval textbook teaches. The output cites paths, not opaque vector IDs. If a query ranks something unexpectedly, you can open the file, see what is indexed, and either edit the doc or refine the query. There is no embedding model whose weights you cannot inspect.

The sleep-product specialist (agents/sleep-product.md) needs no changes — it already maintains knowledge files, feature PRDs, and the tag set. Recall reads what sleep-product already maintains. Same pattern as the snapshot: the deterministic tier curates, the on-demand tier queries.

---

Part of the dreamcontext deep dive — Home · README