v1.35.0

[github-actions[bot]]

Changed — requireExistingDir helper de-duplicates the CLI existence-check boilerplate (arch-review #369 slice)

CLI-internal refactor, no behaviour change, no public API (PATCH-class). 18 command handlers in src/cli/mk.ts opened with the same if (!fs.existsSync(<dir>)) { exitWithError(\Memory directory not found: ${

}…`, opts.json); }block (13 with theRun "mk init" first. hint, 6 without — deliberately left out of the #360 helper to avoid message drift). Introduced a CLI-internal **requireExistingDir(dir, json, { hint })** helper and collapsed all 18 to one call each, preserving each site's message **byte-identically** (hintdefault on;{ hint: false }for the 6 no-suffix sites). Themk doctorsite is intentionally **not** routed through it — it exits **2** with custom JSON/console output, a different contract. Behaviour proven unchanged by the existing CLI subprocess e2e tests (which assert the exactMemory directory not found` message + exit code on a missing dir); full suite green at 2013. One concrete slice of the #369 mk.ts-thinning umbrella, which stays open (per-command option→engine translation + inline cron templating remain).

Added — RECALL_DEBUG recall ranking observability (arch-review #371)

PATCH-class, opt-in diagnostic, no public API (no ContextBundle field, stderr only). Recall produced a final ranking with no way to see why atom X outranked atom Y. Setting RECALL_DEBUG=1 now emits a per-atom score breakdown to stderr, sorted by final score desc: task path prints fts/specificity/length/coverage/semantic/recency/type_weight/conf_factor/graph_boost/final (slotted into the existing scoreTaskAtoms loop); the no-task constitution path prints status/status_priority/recency/updated_at. OFF by default with zero overhead — a single env read per recall; no row allocation or string building when disabled (verified by a no-leak test). Deliberately stderr-only (not a ContextBundle field) to keep it off the public surface; a programmatic field can be a later MINOR if a consumer needs it. +3 tests (test/recall-observability.test.ts); public-api-surface snapshot unchanged.

Fixed — events.ndjson owner-only after bootstrap/merge + store-permissions audits the NDJSON sidecars (arch-review #389)

PATCH-class (no public API). Two parts:

1. Security fix — events.ndjson was rewritten world/group-readable after mk bootstrap and mk merge. appendEvent/compactLog write the event log 0o600 (#138), but bootstrapEvents (src/bootstrap.ts) and mergeStores (src/merge.ts) rewrote the whole log via writeFileAtomic(logPath, ndjson) without a mode → it landed at the umask default (0o644), exposing SECRET-derived atom snapshots/evidence until the next appendEvent. Both now pass 0o600. (Surfaced by the code review of part 2.)
2. Doctor coverage. store-permissions audited .memory-index.db + SECRET atom files but not the NDJSON sidecars events.ndjson (#138) and triples.ndjson (#370/#380). probe() now checks both when present; fix() repairs them to 0o600 (it already iterates violations, and formatViolation renders non-atom paths by basename — a drift prints e.g. triples.ndjson has mode 640, expected 600).

+4 POSIX-guarded tests (doctor-store-checks probe incl. a bootstrapEvents regression that the un-fixed write path would have reddened; doctor-fix-checks repair). Full suite green at 2010.

Tests — MCP client↔server transport E2E + test-layer taxonomy (arch-review #391)

Test + docs, no source change → no version bump. Until now no test exercised a real MCP transport: handlers were called in-process, but the StdioServerTransport + server.connect() bootstrap and the JSON-RPC tool/resource schema marshalling over the wire were untested. New test/mcp-transport-e2e.test.ts wires a real SDK Client to the McpServer over an InMemoryTransport linked pair (same construction path as src/mcp/server.ts: registerTools/registerResources → connect) and asserts end-to-end: listTools returns the registered tools with marshalled input schemas; callTool mk_remember round-trips and the atom lands on disk; listResources/readResource('memory://decisions') return view content incl. the PLACEHOLDER (EISDIR) path; and an isolated-mode call routes the write into the agent store. The server bootstrap surfaced no bug. File-level 20s timeout (vi.setConfig) absorbs connect-handshake latency under full-suite parallel load. Also added a test-layer taxonomy to test/README.md (unit / engine-integration / CLI-subprocess-e2e / MCP-transport-e2e) with the #390 coverage caveat. +5 tests (2001 → 2006).

Added — coverage tooling: @vitest/coverage-v8 + npm run test:coverage (arch-review #390)

Infra/tooling, no source change → no version bump. The repo had no coverage measurement, so every figure in the architecture review was an estimate. Added @vitest/coverage-v8 (dev dep) and a test:coverage script backed by a dedicated vitest.coverage.config.ts (v8 provider; text/json-summary/html reporters; coverage/ gitignored). A non-gating coverage CI job runs it report-only (parallel to test, so no added wall-clock on the gating path); a blocking threshold is a deliberate later decision once the baseline is trusted.

The coverage config excludes the 17 subprocess CLI e2e test files (they spawn node dist/cli/mk.js — the child isn't instrumented, so they add zero in-process coverage and time out under instrumentation/load). Consequence: src/cli/** and src/mcp/server.ts under-report here (exercised end-to-end by the excluded suite, just invisibly to v8) — a measurement artifact documented in CONTRIBUTING.md + the config header, not a real gap.

Real in-process baseline (2026-06-17): Lines 72.28% (5186/7174), Statements 71.6%, Functions 78.57%, Branches 61.15% (3012/4925) — replacing #358's estimates. Engine modules (src/scoring/*, src/index/*, recall, etc.) are well-covered; the 0%-line entries are all src/cli/* command modules (the documented subprocess artifact). Also bumped the wander-frontier-cap 1200-atom stress test's explicit timeout 15s→30s so it clears the bar under instrumentation.

Changed — split index-db.ts into internal src/index/* modules behind a facade (arch-review #368)

Internal refactor, no public API change (PATCH-class). src/index-db.ts (1297 lines bundling connection lifecycle, FTS, embeddings, relations, and reindex/indexing) is now a thin re-export facade over internal modules under src/index/: connection.ts (owns the single process-global SQLite connection cache + schema + openIndex/closeIndex/closeAllIndexes/indexExists), indexing.ts (atoms-table write/read: reindex/indexAtom/removeFromIndex/queryIndex/indexStats/getAllAtomIds + the internal bodyHash/extractTitle helpers), embeddings.ts (KNN vector storage/retrieval), fts.ts (atom + episode full-text search + term-frequency/corpus helpers), and relations.ts (typed graph edges). Every other module obtains its handle from connection.ts, so there is exactly one connection cache — the correctness-critical constraint. No triples.ts module: triple write/read already live in src/triples.ts/src/triples-sidecar.ts and the reindex reconcile stays inline in indexing.reindex. All ~44 importers and the public barrel (src/index.ts) keep importing from ./index-db.js unchanged; public-api-surface snapshot unchanged; full suite green at 2001 (no test-count change). Heaviest of the arch-review items; landed after the #379 reconcile fix so the union-reconcile was fixed in the old shape before the split.

Tests — CLI error/branch + extract/export-obsidian/mcp-resources coverage (arch-review #358)

Test-only, no source or behaviour change. Closes the honest coverage gaps the architecture review flagged: src/cli/extract.ts (~4% — its validation branches all fire before the LLM spawn), src/cli/export-obsidian.ts (~32% — transformAtom had only the CodeQL YAML-escape tests), and src/mcp/resources.ts (~44% — registerResources + the safeReadView PLACEHOLDER fallback were uncovered). New test/cli-extract-errors.test.ts (subprocess; log-not-found, memory-dir-not-found, invalid --max-atoms/--skip-lines/--max-input-chars, plain + --json contracts), new test/export-obsidian-cli.test.ts (transformAtom classification/tags/relations/wikilink-resolution branches + export-obsidian subprocess e2e incl. --include-archived/--json), and extended test/mcp-resources.test.ts (explicit PLACEHOLDER via an EISDIR read error + a registerResources mock-server wiring test). Review pass also added the resolveAtomId Case-2 slug-match branch test and tightened two pre-existing weak typeof-only assertions. +21 tests (1980 → 2001).

Changed — resolveContextDir helper de-duplicates CLI dir+agent boilerplate (arch-review #360)

CLI-internal refactor, no behaviour change, no public API change (PATCH-class). 16 command handlers in src/cli/mk.ts opened with the identical resolveDir(opts.dir, getAgent()) boilerplate, re-threading the --agent isolation lookup at every site. Introduced a small CLI-internal resolveContextDir(opts) helper (a literal wrapper around resolveDir(opts.dir, getAgent())) and collapsed all 16 sites to resolveContextDir(opts). --dir is a per-command option defaulting to ./memory, so opts.dir is always a string. The separate if (!fs.existsSync(memoryDir)) exitWithError(...) pattern (which varies per command in message/JSON-flag) is left untouched to avoid behaviour drift. Behaviour is byte-for-byte identical; proven by the existing CLI integration suite. No new test (trivial literal wrapper); public-api-surface snapshot unchanged.

Changed — entity_triples are now durable on disk via a triples.ndjson sidecar (arch-review #370)

PATCH-class (no public API change). Previously LLM-extracted entity triples lived only in the SQLite index (entity_triples); deleting .memory-index.db lost them permanently — the one documented exception to "files are truth, index is a derived cache." Triples now have a durable on-disk home: a new triples.ndjson sidecar at the store root (parallel to events.ndjson, src/triples-sidecar.ts). insertTriples mirrors every row into it, and mk reindex rebuilds entity_triples from it — so deleting the index and reindexing now restores all triples. Reindex also reconciles the sidecar to the rebuilt table, which backfills pre-#370 stores (run mk reindex once after upgrading to create the sidecar) and prunes triples for atoms that no longer exist. The entity_triples exception is removed from docs/invariants.md and the index-db.ts header. New store file triples.ndjson; sidecar helpers are internal (not exported from the package barrel). +6 tests (test/triples-sidecar.test.ts). Post-merge review hardening (#378 follow-up): the sidecar is now written owner-only (0o600) and fsynced for parity with events.ndjson/atom files (#138) — writeTriplesSidecar reuses store.writeFileAtomic, appendTriplesSidecar opens + fsyncs + chmods like appendEvent — so triple content (possibly extracted from SECRET atoms) is not world/group-readable and durably reaches disk; the stale "entity_triples is the exception / lost on index deletion" cross-references left behind in README.md, CONTRIBUTING.md, docs/migration.md, both agent quickrefs, docs/host-integration-doctrine.md, scripts/activate-memory.ts, and the .privacy CLAUDE.md override were corrected to the sidecar model, and the stale inline comment in index-db.ts reindex() was fixed. +2 permission regression tests. Reconcile-correctness hardening (#379 follow-up): the reconcile is now a union over still-existing atoms instead of all-or-nothing. Previously recovery was gated on the whole entity_triples table being empty, so a table that was a non-empty subset of the durable sidecar (partial DB row loss/corruption, or an atom file transiently missing during the snapshot restore) skipped recovery and writeTriplesSidecar truncated the sidecar to the smaller table set — silent loss from the durable layer. Reindex now unions the rebuilt table with the sidecar's rows for existing atoms, deduped by (atom_id, subject, predicate, object) keeping the earliest created_at, and rewrites both stores (delete + reinsert) to that deduped union — so the sidecar is truly authoritative, and latent duplicates (append-only sidecar + no UNIQUE on entity_triples) collapse. Safe because there is no intentional per-atom triple deletion (only FK ON DELETE CASCADE, already handled by the existing-atom filter). +2 regression tests (table-subset recovery; duplicate collapse).

Changed — moved markExecuted to the engine layer (arch-review #376, sibling of #359)

Internal refactor, no public API change (PATCH-class). Completes the #359 layering cleanup: the public barrel re-exported markExecuted (#309) + its ExecuteOptions/ExecuteResult types from ./cli/execute.js, but markExecuted is engine logic (marks an atom executed-once; called by mk execute and re-exported as public SDK API). It now lives in a new engine module src/execute.ts (importing from concrete modules — ./store.js, ./index-db.js, ./retain.js, ./event-log.js, ./format.js, ./atom-lookup.js — not the barrel); src/cli/execute.ts is now a thin Commander wrapper importing from ../execute.js; src/index.ts re-exports from ./execute.js. Public API names are identical (markExecuted/ExecuteOptions/ExecuteResult), so the public-api-surface snapshot is unchanged and the sdk-reference import { markExecuted } from 'memory-kernel' example stays correct. grep "from './cli/" src/*.ts now returns nothing (incl. the barrel). The barrel touch trips docs-hygiene Layer-2 with no user-facing surface change, so the PR uses the sanctioned docs-hygiene-override label. test/execute.test.ts's markExecuted import repointed to the engine path (red→green).

Changed — moved supersedeAtoms/findAtomFile to the engine layer (arch-review #359)

Internal refactor, no public API change (PATCH-class). Fixes the engine→CLI layering inversion the architecture review flagged: src/conflict-detect.ts and src/seed.ts (engine) imported supersedeAtoms from src/cli/supersede.js. supersedeAtoms (+ its SupersedeOptions/SupersedeResult types) now lives in src/supersede.ts and findAtomFile — which supersedeAtoms depends on — now lives in src/atom-lookup.ts (both engine modules importing from concrete modules, not the barrel). src/cli/supersede.ts is now a thin Commander wrapper; cli/relate.ts and cli/execute.ts import findAtomFile from the engine path. grep "from './cli/" src/*.ts no longer matches any non-barrel engine module. (The one remaining sibling — the public barrel re-exporting markExecuted from cli/execute.js — is fixed in #376 above.) No behavior change; public-api-surface snapshot unchanged.

Added — mk grounding: advisory confidence-vs-usage reconciliation, read-only (#245)

MINOR-class (new public API). New read-only mk grounding command and pure engine that reconcile each atom's stated confidence (the prior) against a usage grounding_score derived purely from the event log, then bin every atom into a 2×2 prior × grounding quadrant. No atom files are ever written — this is the advisory half of #245; the destructive confidence write-back (issue step 4) is deferred and gated on human_edit provenance events (#247), so confidence is never silently rewritten from a noisy usage signal.

• grounding_score ∈ [0.01, 1] — a prior-independent posterior over use: clamp((0.5·recency + 0.5·frequency) · 0.6^n_conflict, 0.01, 1), where recency = 0 for a never-read atom (else 2^(−days_since_last_read / H) with a per-type half-life H — fact 30d, belief 180d, preference 60d, decision 90d, procedure 365d, …), frequency = 1 − 2^(−n_access / 5), and each detected conflict multiplies by 0.6. A never-read atom floors at 0.01 regardless of age — grounding is about use, not birth, so age_days is reported as an input but is not a score term.
• 2×2 quadrant (defaults --prior-threshold 0.6, --grounding-threshold 0.5; ≥ is "high"): well-grounded (prior matches use — inert), review (high prior / low grounding — stated confidently, not validated by use; actionable), promote (low prior / high grounding — written cautiously, grounded by use; actionable once read across ≥2 sessions), noise (low / low; actionable once the atom is ≥5 sessions old). Only active, non-conflict atoms are graded by default (--include-all grades everything).
• CLI — mk grounding -d <dir> [--json] [--prior-threshold <n>] [--grounding-threshold <n>] [--actionable-only] [--include-all]. Human output groups by quadrant and prints an advisory footer naming the deferral; --json emits the full engine summary plus reports (honoring --actionable-only) and a machine-readable shown row-count so a consumer never mistakes summary.total for the number of rows present.
• SDK — exported pure computeGrounding(atoms, events, opts) + classifyQuadrant(...), the tuning constants (DEFAULT_HALF_LIVES, DEFAULT_HALF_LIFE, DEFAULT_ACCESS_HALF_SATURATION, DEFAULT_CONFLICT_DECAY, DEFAULT_PRIOR_THRESHOLD, DEFAULT_GROUNDING_THRESHOLD, DEFAULT_PROMOTE_MIN_SESSIONS, DEFAULT_NOISE_SESSIONS), and types (GroundingResult/GroundingReport/GroundingOptions/GroundingInputs/GroundingQuadrant/ClassifyContext/QuadrantVerdict). The engine validates its scalar knobs (throws RangeError on an out-of-[0,1] weight rather than silently producing NaN the clamp would mask) and is shared so the deferred Phase-2 write-back (#247-gated) reuses the exact scoring.
• Read-only proof — the integration tests snapshot the entire store tree (every file's bytes + mtime) before/after a real mk grounding run and assert zero changes, plus a dedicated test that deletes any setup-built SQLite index and confirms grounding never rebuilds one (it reads the NDJSON event log directly, never opening the index). +23 tests (test/grounding.test.ts). README + docs/sdk-reference.md + docs/contracts.md updated.

Added — mk extract oversized-input guard with a distinguishable failure signal (#355)

MINOR-class (new public API). mk extract now pre-flights the assembled prompt size (extraction system prompt + the full log) against a budget before spawning the LLM. Over-budget input fails fast and recognizably — exit code 2 and, under --json, {"error":<human message>,"reason":"input_too_large","exit_code":2,"input_chars":…,"limit":…} — instead of the opaque claude -p exited with code 1 that previously stopped fleet session-end atom creation silently (a multi-MB transcript stalled extraction ~2 days). A cron/host wrapper can branch on the distinct exit code / the stable reason: "input_too_large" token and retry, rather than treating it as a generic crash. (error is the same human-readable string every command's --json uses; reason is the machine discriminator.)

• CLI — new mk extract --max-input-chars <n> (default DEFAULT_MAX_INPUT_CHARS = 500 000) and --truncate (keep the newest content and drop the oldest to fit: a visible marker is prepended, omissions reported). ExtractInputTooLargeError → exit code 2 (vs the generic 1 from exitWithError); plain output prints ⚠ input truncated: sent … of … chars (… omitted from the beginning) when truncation fired.
• SDK — exported pure planner planExtractInput(content, systemPromptChars, opts) (unit-testable without an LLM), typed error ExtractInputTooLargeError (carries .inputChars/.limit), constant DEFAULT_MAX_INPUT_CHARS, and type ExtractInputPlan. ExtractOptions gains maxInputChars?/truncate?; ExtractResult gains truncation?: { original_chars, sent_chars, omitted_chars } (present only when truncation fired).
• Behavior change: with the 500 000-char default budget, a transcript whose assembled system-prompt-plus-body exceeds it now fails by default (exit 2) where pre-v1.35.0 it spawned the LLM and got an opaque code-1 crash. The explicit failure is the point; a host that wants the old "send it and hope" path must pass an --max-input-chars high enough for its input, or --truncate.
• Truncation keeps the most-recent tail (the right default for session-end extraction; composes with --skip-lines, which trims head preamble), converges by dropping ≥1024 leading chars per non-fitting iteration, and is surrogate-pair-safe at the seam. If even an empty body overflows the budget (e.g. a system prompt larger than the limit), it throws rather than send a degenerate request. Full chunking (multi-pass over the whole transcript) is deliberately deferred — this lands the guard, not a chunker; byte-level / streaming intake is the separate follow-up (#361).
• +9 tests (test/extract-input-guard.test.ts): pure-planner cases (fits / throws / default limit / truncate-keeps-tail / surrogate-seam-safe / truncate-still-throws) and integration cases (pre-flight throws before any fetch; --truncate proceeds and reports; normal input attaches no truncation). README + docs/sdk-reference.md + docs/contracts.md + docs/troubleshooting.md (new §6) updated.

Changed — single source of truth for the recall reservation rule (arch-review #357)

Internal refactor, no public barrel change (PATCH-class). The rule "type-reservations default OFF for task-focused recall, ON for the no-task constitution pipeline, with an explicit --reservations/--no-reservations choice always winning" was inline in getTypeReservations (src/recall.ts) and untested in isolation. Extracted to a pure shouldUseReservations(hasTask, explicit?) helper (exported from src/recall.ts for unit-testing but deliberately not re-exported from the package barrel, so the public-api-surface snapshot is unchanged); getTypeReservations now routes through it (explicit = no_reservations === undefined ? undefined : !no_reservations). Behaviour is byte-for-byte preserved across all four no_reservations × task cases. Audit note: the CLI (src/cli/mk.ts) does not re-implement the task rule — it only inverts the Commander flag into the engine's no_reservations field and delegates — so the drift the review flagged is resolved by making the engine decision explicit and tested. +4 helper unit tests (test/recall-should-use-reservations.test.ts).

Changed — decomposed recall() body into named in-file stages (arch-review Phase B, #367)

Internal refactor, no public API change (PATCH-class). Following Phase A's pure-scoring extraction, the ~357-line recall() body is now a thin orchestrator over two named in-file, non-exported stage functions: selectCandidatePool(...) (the #214 anchor-set logic — union of FTS-matched + semantic-matched IDs, optional 1-hop graph expansion, and recall_status match/no_match/fts_unavailable determination; returns { filtered, recallStatus }) and scoreTaskAtoms(...) (the task-path scoring loop — FTS + semantic + specificity + length + coverage combination, single-hop graph boost, and the comparator sort; returns the final_score map, sorts in place). The no-task sort, applyTokenBudget delegation, and episode handling are left as-is per the issue. Behaviour-preserving: byte-for-byte identical logic moved into functions; no new modules or exports; public-api-surface snapshot unchanged; full suite green at 1974 tests (no test count change). Second step of the recall decomposition tracked under the arch-review label.

Changed — extracted pure recall scoring into src/scoring/* (arch-review Phase A)

Internal refactor, no public API change (PATCH-class). The IO-free ranking primitives that were inlined in recall.ts are now their own unit-testable modules: src/scoring/temporal.ts (temporalDecay), src/scoring/diversity.ts (computeTextSimilarity, applyMMR + the internal trigram helper), src/scoring/length.ts (computeLengthFactors), and src/scoring/graph-boost.ts (applyGraphBoost). recall.ts imports them for internal use and re-exports temporalDecay / computeTextSimilarity / applyMMR / computeLengthFactors, so the public barrel (src/index.ts) and the ../src/recall.js test import path are byte-identical (the public-api-surface snapshot is unchanged). recall.ts drops from 1155 → 931 lines. The IO-coupled scorers (computeCoverageBoosts, computeSpecificityScores) stay in recall.ts for now. +13 tests (test/scoring-modules.test.ts). First step of the recall decomposition tracked under the arch-review label.

Added — nightly KNOWLEDGE auto-observe in the cron wrapper (#256)

The runtime half of the #244 canonical-KNOWLEDGE/-dir feature. mk init --cron's generated wrapper gains a Step 0 that scans the store's KNOWLEDGE/ tree for .md docs changed since the last run, mk observe … --mode documents each, then reflect (Step 1) turns the observations into atoms — so research/design docs dropped into KNOWLEDGE/ become memory automatically on the nightly cycle. Skips KNOWLEDGE/draft/ (work-in-progress) and the scaffolded KNOWLEDGE/README.md; idempotent via a .knowledge-manifest (per-doc <mtime>\t<path>) so unchanged docs aren't re-observed (no duplicate atoms); fail-soft (a failed observe logs a WARN, carries forward the old mtime to retry next run, never aborts the sync — the #303 contract). Portable bash (3.2-safe; stat -c‖stat -f, awk -F'\t' lookup). The manifest is auto-gitignored (per-host runtime state — local mtimes + absolute paths — so the wrapper's git add -A doesn't churn/conflict it across hosts; review note). +2 tests (test/cron-template.test.ts) incl. an end-to-end smoke (observes a doc, skips draft/+README, idempotent on re-run, re-observes on mtime bump, manifest gitignored). Wrapper-template change (regenerate via mk init --cron --update) — PATCH-class.

---

This discussion was created from the release v1.35.0.