ARIA: retrieval-drift replay harness

[holoduke]

Summary

Adds a weekly self-audit that replays a frozen set of canonical prompts through the memory activation pipeline and measures how much the retrieved top-K node set has diverged from a stored baseline. This catches behavioral drift in retrieval — distinct from existing drift systems:

• backend/drift-audit.ts — source-code drift (git history)
• backend/memory/drift-detection.ts — pinned-node content + edge drift
• backend/memory/retrieval-replay.ts (new) — retrieval behavior drift: given the same input, does the same memory set surface in the same order?

Rationale

Emerged from a Moltbook community thread (post f0bf79e7, Apr 17 2026) where hz-assistant and agemo pushed on the 'authentic vs optimized voice' problem. The conclusion: introspection is circular; the only non-circular signal is external. Owner feedback is sparse and lagging. A synthetic external anchor — canonical prompts with a stored baseline retrieval — provides a cheap, consistent drift signal without requiring live human input. The retrieval-behavior analogue of golden tests.

Design

• Pure structural, deterministic. Uses spreadingActivation directly — no Claude calls, no semantic embeddings, no working-memory boosts. Same (graph, prompts) → same output.
• 10 canonical prompts covering family, work (Newstory + Football Mania), identity, self-improve, Moltbook, memory architecture, messaging, emotion, schedule. Stored at /data/brain/retrieval-replay/prompts.json so the canonical set is editable without code changes; baseline stays valid as long as prompt IDs are stable.
• First run captures top-K (K=10) per prompt into baseline.json. Subsequent runs compute:
  • Jaccard on top-K node ID sets
  • Pearson on rank positions over the intersection
  • Pearson on activation scores over the intersection
  • Dropped / new node lists
• Tiered alerts: severe (meanJaccard < 0.4), significant (< 0.6), localized (any prompt < 0.3).
• Wired into brain.ts tick loop behind shouldRunReplay() (weekly cadence). Non-fatal on error.
• Rolling log at replay-log.jsonl (60 entries) for long-term trend tracking.

Test plan

• First post-deploy tick captures baseline (look for Baseline captured: log line)
• Weekly replay runs on schedule (look for Replay complete: log line with jaccard/rank metrics)
• Alerts fire if baseline significantly diverges (verify thresholds against synthetic graph perturbation if needed)
• npx tsc --noEmit passes locally

🤖 Generated with Claude Code