perf: cache MemoryEngine vector matrix instead of reloading per recall

[saucam]

Summary

MemoryEngine.recall() loads the full embedding matrix from SQLite on
every call (loadVectorMatrix selects every embedding row, allocates
fresh Float32Arrays, then linear-scans). With memory enabled this is
in the hot path of every Claude turn that uses the recall tool — each
invocation re-allocates the full matrix and re-scans it.

Why it matters

• At the user's current scale (single user, low thousands of episodes)
  it's invisible — ~16 ms per recall on a 10k-episode workspace.
• Above ~10 k episodes the per-recall cost climbs linearly and
  allocations dominate (every call allocates new typed arrays).
• It's also the kind of cost that compounds with sub-agent fan-out:
  one parent turn spawning 3 reviewer agents that each hit recall
  pays 4× the matrix walk in serial.

Proposed fix

• Keep an in-memory Map<workspaceId, Float32Array> cache on
  MemoryEngine.
• Invalidate by tracking ingest count or a workspace-level version
  stamp; bump on each applyMessage / applyDelta that produces a
  new episode.
• Reuse the cosine loop against the cached matrix.

Out of scope

Approximate-nearest-neighbor (HNSW, faiss, sqlite-vec's KNN) is the
real fix beyond ~100 k episodes — leave it for when scale actually
demands it.

Source pointers

• src/daemon/memory/engine.ts:126-137
• src/daemon/memory/store.ts:605-620

Severity

P2. Quality-of-life perf for power users; not blocking anyone today.
Surfaced in the 2026-05-08 daemon audit.

🤖 Generated with Claude Code