macrodata@0.11.0

Minor Changes

• #82 23fc789 Thanks @jasikpark! - Ambient recall now keeps its index current on its own. The recall worker reconciles the corpus at startup (which is also the first-run build), and a new hook queues a reindex at SessionStart and after any Write/Edit under entities/ or journal/ or any macrodata journal tool call. The hook only drops a request into the worker's mailbox, so it never loads a model, and the one long-lived worker serializes every index write. SessionStart prints a one-line notice while no index exists yet.

  The index now records which embedding model built it. When the worker finds an index it cannot use (unparsable, or built by a different model), it moves the file aside and rebuilds once; if the rebuilt index is unusable too, it stops reindexing and SessionStart says why until the index is replaced. A reconcile that stops making progress restarts the worker, the same way a stalled search does.

macrodata@0.10.0

Minor Changes

• #81 4316bc8 Thanks @jasikpark! - Ambient recall reconciles its index incrementally instead of re-embedding the whole corpus.

  reconcileCorpus and reconcileSource compare each projected item against the indexed one:
  items whose content already has a stored vector reuse it (a moved timestamp or category, a
  rename, a journal line shifted by an insert above it), only genuinely new content is embedded,
  and unchanged items are skipped. A rename reconciled path-by-path reuses the vectors the first
  path's deletion just pruned. A pass with nothing to do leaves index.json untouched, and an
  interrupted pass resumes where it stopped.

  Deletion follows the projection's authority. An id the scan no longer produces is pruned unless
  its source lies under something the scan failed on — a root or directory that failed to list, an
  entry that could not be stat'd, or an unreadable, malformed, or symlinked source — or under a
  missing journal or entities root. A file with a size but no allocated blocks (evicted by iCloud
  or another sync provider) or one rewritten during the read counts as unread, not empty. An
  unparsable final journal line with no trailing newline (an append in flight, or a truncated
  rewrite) is not a malformed record, and the vectors indexed at or past it are kept.
  reconcileSource treats a path that is gone (ENOENT under a root that still exists) as a
  confirmed deletion of its source, or of every source under it when it was a directory. A path
  the scan would not index — a symlink at any depth, a spelling that differs from the on-disk
  name, a dot path, a wrong extension — changes nothing.

  Index writes are batched and atomic: each commit writes a temp file and renames it over
  index.json, so an interrupt mid-commit can no longer leave a truncated index that every later
  run fails to parse. A commit refuses to overwrite an index.json that another process committed
  after this one loaded it, instead of silently reverting that write. Leftover temp files from a
  killed run are swept on load. An update's changes stay invisible to searches until it commits,
  and replacing a vector recomputes its norm. Writers in one process are serialized.

  bin/recall-reindex.ts now reconciles by default; --full re-embeds everything (for an
  embedding-model change) and --prune-only deletes only what the current corpus proves is gone. An unparsable index.json
  makes every mode fail with a message naming --full, which moves it aside and rebuilds.
  Unknown or conflicting flags exit with a usage message.

Patch Changes

• #79 f3270f7 Thanks @jasikpark! - One canonical projection of the memory corpus, shared by the MiniLM and Qwen indexers.

  Both indexers carried their own copy of the journal and entity parsers, so the two
  indexes could disagree about what a source is and which units it produces. There is now
  a single projection — projectJournalFile, projectEntityFile, scanCorpus — and both
  consume it. A source is its path relative to the journal or entities dir, so an entity
  nested under a category (entities/people/team/bob.md) carries its full
  category-relative stem in both source and its ids; the old basename-derived ids
  collided with an immediate sibling of the same name. Nested entity files are indexed on
  a full rebuild now, where only the incremental daemon path saw them before, so the first
  shared rebuild re-embeds them once.

  A scan now knows whether it is authoritative, and reconciliation depends on it. A file
  that cannot be read, a directory that cannot be listed, and a journal line that is not
  an object with string topic and content all mark their source incomplete while
  keeping whatever was readable — "5", null and [1,2,3] parse cleanly and used to
  index as [undefined] undefined. An incomplete scan never prunes: its missing items
  would otherwise read as deletions and delete live vectors for sources that were merely
  unreadable. A complete scan is trusted even when it is empty, so a wiped corpus
  converges to an empty index instead of retaining every stale vector forever; the one
  exception is both roots missing while the index holds vectors, which is a misconfigured
  MACRODATA_ROOT far more often than a deliberate wipe.

  Symlinks are never followed into the corpus. A link named like a corpus file — say
  entities/people/notes.md pointing at ~/.ssh/id_rsa — would otherwise have its target
  read, embedded, and left retrievable through search_memory and ambient recall. The
  refusal is enforced where the bytes are read (projectEntityFile, reached by the
  daemon's live-watch reindex as well as the batch walk) and the walk decides on lstat
  rather than the dirent type bits, which some filesystems report as unknown. Schedule
  files get the same treatment: followSymlinks: false on a chokidar watcher governs
  chokidar's own traversal and has no effect on readFileSync, so the reminders handlers
  refuse a symlinked schedule before reading it rather than injecting whatever it resolves
  to into the next session's context.

  Also: an entity file directly at the entities root has no category to take its type from
  and is refused rather than indexed under a type named after the file, and the embedding
  input is cut on whole characters, so a note with an astral character at the 2000-char
  boundary no longer embeds a lone surrogate.

macrodata@0.9.3

Patch Changes

• #74 04f66b5 Thanks @jasikpark! - Remove the unused @tobilu/qmd runtime dependency. Macrodata does not import the
  QMD SDK; it remains prior art for future index work rather than install-time code.

macrodata@0.9.2

Patch Changes

• #72 ab3e37f Thanks @jasikpark! - The distill skill's per-transcript sub-agents now run on sonnet explicitly. An
  unpinned Task spawn inherits the parent run's model, so a memory-maintenance schedule
  on opus put every extraction worker on opus too. Extraction over an already-filtered
  transcript does not need it; the parent's state-file rewrite pass is where the heavier
  model earns its keep.

macrodata@0.9.1

Patch Changes

• #67 272a6c4 Thanks @jasikpark! - Recall worker hardening, following the hook-managed worker that shipped in 0.9.0.

  The pidfile-claim guard now asks the same question as the ps classifier: is this
  process a worker for this state root? It used to accept any process carrying the
  worker sentinel, so a claim held by a neighbouring root's worker — reachable through
  pid recycling after a reboot — survived every pass: the guard saw a worker and left
  the claim, the classifier saw none for this root and spawned one, and the fresh
  worker found the claim held by something alive and stood down. Recall stayed dead in
  a state no later pass could undo. One shared predicate now, so there is no second
  copy to drift.

  A pipeline that never settles is now survivable. It used to be worse than a crash: the
  worker stayed in ps holding the claim, read as healthy, and made every replacement
  stand down while dropping every request. The pipeline now runs under a deadline and the
  worker exits when it misses one, so the next pass can start a worker that serves —
  20 minutes for a first run that may download models, 2 minutes once they are loaded
  (MACRODATA_RECALL_WEDGE_COLD_MS, MACRODATA_RECALL_WEDGE_WARM_MS). A spawn that dies
  during startup is no longer silent either: two consecutive spawns that leave no worker
  now say startup is failing instead of logging another clean first start. That count is
  deliberately not a time window — a window between attempts measures how fast you type,
  and at any real prompt cadence every gap exceeds it.

  Also: mailbox files left by sessions that have ended are swept hourly past a 7-day TTL
  (only request- files were ever consumed, so the directory grew for the life of the
  state root and the worker's 5s sweep read all of it), a request file the worker cannot
  parse is quarantined instead of left to be found again by every one of those sweeps for
  the life of the process, the recall logs are trimmed to their last 512KB past 1MB,
  and a root in config.json that isn't a string falls back to the default in both the
  shell and TypeScript resolvers rather than resolving to a bogus path in one and throwing
  in the other.

  Behavior change worth knowing: when a hand-started worker is running on a state root,
  the hook now reaps the installed workers on that root. Every worker drains the same
  mailbox, so leaving both up made each request a race between two copies of the code.
  The hand-started worker is still never killed.

  The state root is now canonicalized the same way on both sides. It is an identity and
  not only a path — the hook writes it into the worker's argv and later finds that worker
  again by comparing strings — so ~/store and ~/store/ were two identities sharing one
  mailbox, each session reading the other's claim as foreign, deleting a live pidfile and
  starting a second worker. Trailing slashes are stripped and an existing directory is
  resolved through symlinks, so the two spellings converge. A root containing a control
  character now falls back to the default instead: ps renders a newline as \012, so
  such a worker can never match its own argv again and every prompt starts another one,
  without bound.

  Text injected into the model's context is neutralized wherever it comes from, the
  detected-user block and the recall status line included. Store content could otherwise
  close the wrapper tag around it and have the rest read as the hook's own output. The
  neutralizer is one shell function now rather than a pattern repeated per site, which
  also removes a portability trap: from bash 5.2 an unescaped & in a ${var//…/…}
  replacement expands to the matched text, so the escaping silently produced different
  results on macOS's bash 3.2 and on CI.

  A reranker that fails now yields no hits rather than hits ordered by the fusion score.
  The relevance floor and the calibration log are both defined in the cross-encoder's
  scale, so passing along numbers from a different one would have read as a working
  recall that had quietly stopped ranking.

  Also: a request the worker cannot parse is quarantined under a name carrying the time it
  was quarantined, so a second unparseable request in the same session no longer overwrites
  the first; injected- records are pruned on every prompt rather than only on prompts that
  inject something; a claim held by a pid that has exited is cleared, and "no such process"
  is now told apart from "not permitted to signal it", which previously read the same;
  and a recall budget larger than setTimeout can hold is clamped instead of overflowing
  into firing immediately, which would have declared every request wedged on arrival.

• #69 9e44589 Thanks @jasikpark! - Throttle the tools-hint hook to fire every 11 turns instead of every turn.

  The hint nudges the model to call recall tools intentionally, but injecting it on
  every single prompt adds context noise for minimal benefit — the model doesn't need
  the reminder on turn 2 if it saw it on turn 1. A per-session counter file in /tmp
  (keyed on CLAUDE_CODE_SESSION_ID) tracks invocations and suppresses output on
  non-interval turns; the first turn of each session always fires.

macrodata@0.9.0

Minor Changes

• #64 3d2e7e9 Thanks @jasikpark! - Add red-flag surfacing channel (state/flags.md), atomic daemon pidfile acquisition, and heartbeat-first arbitration for scheduled skills.

  Red-flag surfacing — scheduled runs can discover issues that never reach the user because they terminate in the model's context. state/flags.md is the new cross-session channel: the daemon fires a macOS notification when new 🔴 items appear, and the prompt-submit hook injects a relay instruction once per session (keyed by session_id + section hash, so every session is reminded and a changed section re-fires everywhere).

  Atomic pidfile — the daemon's existsSync check followed by a plain writeFileSync let two daemons started in the same instant both survive the guard, double-firing every cron. The pidfile is now acquired with writeFileSync(..., { flag: "wx" }); on collision the holder is liveness-checked, a stale file is unlinked and the acquisition retried once.

  Heartbeat arbitration — dreamtime and memory-maintenance now open by banking a journal heartbeat claiming the run, then re-reading to arbitrate, so if a double-fire does happen the losing twin stands down instead of both writing state.

  Ported from ascorbic/macrodata PRs #30 and #37.

• #65 bbfad4b Thanks @jasikpark! - Replace session/subagent reminder delivery with a deterministic notify mode.

  • schedule now offers two delivery modes: notify (default) and headless. The session mode — claim files drained into active sessions as background subagents — is removed; stored schedules with delivery: "session" fire as notify automatically.
  • notify runs no model: at fire time the daemon posts a macOS notification and upserts a - [id] fired <time> — <payload> line into state/reminders.md (a re-fire replaces the schedule's own line). Reminders surface in sessions via the SessionStart compose hook and a prompt-submit relay nudge; removing the line with the Edit tool clears the reminder.
  • headless is unchanged: a detached claude --print on the tick.
  • Schedule hardening: a schedule's identity is its reminders/<id>.json filename — the id in the body is ignored for job keys and deletes, and remove_reminder refuses any id outside [A-Za-z0-9_-]{1,64}, so neither path can be aimed at a file outside reminders/. A one-shot whose date doesn't parse (or already passed) is refused by schedule with Not scheduled: … instead of being saved and silently deleted. Editing a schedule file re-arms its job (the old job kept firing the old fields). Firing runs under a guard, so a payload that breaks the notification (a NUL byte, no text at all) logs an error instead of exiting the daemon.
  • Relay hardening: the prompt-submit reminder and red-flag relays keep whole lines within a 2,500-byte budget and end with a … N more line(s) not shown; read state/<file> marker, so an oversized section can't overflow Claude Code's hook-output cap and erase every block with it. The ## ⏰ heading in state/reminders.md is no longer load-bearing: the relay and the SessionStart composer key on - entry lines (a heading-only file composes nothing), and the daemon restores the heading if a hand-edit removed it.

Patch Changes

• #62 13f8e25 Thanks @jasikpark! - Manage the ambient-recall worker from macrodata-hook.sh, on both hook events, so a
  plugin update takes effect without waiting for a new session. The worker had its own
  SessionStart-only supervisor, and SessionStart does not fire on /plugin update +
  /reload-plugins — so the pass that reaps the previous version's worker only ran once
  a session happened to open, and until then a freshly installed release kept serving
  recall from the old cached code. The daemon already converged on every prompt; the
  worker now does too, through the same verified-kill path. Per-prompt passes stay silent
  unless they act, so neither the model's context nor the log gets a line per message.

  Converging on every prompt also means concurrent sessions can observe the same
  worker-less window and spawn into it together, so the worker now claims
  .recall/worker.pid before it can load a model and stands down if another process
  already serves that state root. A claim whose process is gone is taken over rather
  than obeyed — the hook stops a stale-version worker with SIGKILL, which never gets to
  clean up after itself. A reboot restarts the PID space from the bottom, where a
  surviving claim can name an unrelated live process and mute recall for good, so the
  hook reads the holder's own command line and clears the claim unless that process is
  itself a worker.

macrodata@0.8.1

Patch Changes

• #60 2e9f122 Thanks @jasikpark! - Roll the ambient-recall worker on plugin upgrade. The supervisor identified its
  workers by state root alone, so a new version adopted the previous version's
  running worker and logged it as healthy — an installed release could serve recall
  from code it does not contain, with no visible symptom. It now classifies each
  worker by the source path in its argv: this version's stays up, another plugin
  version's is reaped (SIGTERM escalating to SIGKILL, then verified) and respawned,
  and a hand-started dev worker keeps running but is announced instead of passed
  over in silence.

macrodata@0.8.0

Minor Changes

• #58 833d5eb Thanks @jasikpark! - Ship ambient recall as part of the plugin instead of a sidecar checkout.

  The retrieval pipeline (Qwen3-Embedding-0.6B / 1024-dim via node-llama-cpp) moves
  into src/recall/, its entry points into bin/recall-{hook,supervisor,reindex,search},
  and its hooks are registered in plugin.json — so a marketplace install gets ambient
  recall with no manual settings.json wiring.

  Runtime state now resolves through the shared getStateRoot() and lives under
  <root>/.recall/ (index, per-session mailbox, calibration, access log, worker logs).
  Previously it was written next to the source, which only worked for a fixed checkout
  path: plugins install into a per-version cache dir, so a source-relative index would
  be orphaned on every release. The leading dot keeps it inside the state root's
  existing "dotfiles are runtime, plain dirs are memory content" ignore rule.

  This also corrects the data root for anyone who is not the original author — the
  sidecar hardcoded ~/Documents/macrodata rather than honoring MACRODATA_ROOT and
  ~/.config/macrodata/config.json.

  Ambient recall keeps its own index: it embeds at 1024 dimensions while the MCP server
  uses MiniLM at 384, so the two cannot share a Vectra store.

  Upgrading from a hand-wired sidecar: kill any worker started by the old supervisor once
  (pkill -f recall/worker.ts) and drop the recall entries from settings.json. The new
  supervisor identifies its workers by an argv sentinel rather than by script path — which
  is what lets it reap the previous plugin version's worker on every future update — so a
  worker predating this change is invisible to it and would keep draining the same mailbox
  alongside the new one.

macrodata@0.7.6

Patch Changes

• #50 d971886 Thanks @jasikpark! - spike(ambient-recall): fix the worker never seeing a request. Bun's fs.watch(dir) on macOS does not deliver an event under the final name of a tmp → rename publish, and both hook request-writes publish that way (atomicWrite), so the worker's filename-matched watch callback never fired for a real request — since the spike's first commit. Recall worked anyway because ingest()'s own unlinkSync is a final-name event, which re-entered the callback and picked up whatever had arrived meanwhile: a self-sustaining chain that lasted only while requests arrived faster than the ~5s rerank, and left the session permanently deaf after the first lull. The watch now ignores the reported filename and re-scans the directory (50ms debounce) on any event, with a 5s interval backstop for a dropped or coalesced FSEvents batch, and drops requests older than MACRODATA_RECALL_MAX_REQ_AGE_MS (default 10 min) for one log line instead of a rerank whose inbox nobody will drain.

macrodata@0.7.5

Patch Changes

• #47 3c1790f Thanks @jasikpark! - spike(ambient-recall): structured worker logging via LogTape. The worker now emits NDJSON records with per-line timestamps under subsystem categories (recall.worker / recall.ingest / recall.pipeline), and the previously-silent paths are visible: a pipeline-start line (a never-settling pipeline is now provable from the log instead of inferable from absence), a warning when the short-search guard drops an already-consumed request, and a queued-behind-active-drain line that surfaces the drain-wedge failure mode in real time.

• #49 e175304 Thanks @jasikpark! - Structured logging via LogTape across the plugin proper. Library modules (indexer, conversations, embeddings, rerank) now log NDJSON records under per-module macrodata.* categories, routed by whichever entrypoint configured a sink: the MCP server sends diagnostics to stderr, and the daemon appends them to .daemon.log (which also captures indexer/conversations records that previously vanished into the daemon's discarded stdout). Fixes a protocol bug: manage_index rebuild/update completions were console.logged onto the MCP server's stdout, which is the JSON-RPC channel. In unconfigured processes (hook scripts, tests) records drop silently, so model-load and index chatter can no longer leak into hook output.