feat: implement Memory v2

[woai3c]

Summary

This PR replaces the sparse legacy auto-memory path with Memory v2: a global, Markdown-first user memory system with durable post-turn extraction and hybrid retrieval.

• Extracts memory once after a root agentLoop finishes normally, using only that turn's bounded projection.
• Persists work through an on-disk job queue, so the answer does not wait for the memory model and interrupted jobs survive restarts.
• Stores full facts in memory/topics/*.md; derives the compact MEMORY.md core profile from those topics.
• Recalls details with exact/alias/path matching and BM25F first, then uses a bounded semantic selector only for ambiguous historical queries.
• Supports cross-repository user memory, conflict replacement, explicit forgetting, manual reload, diagnostics, and late recall from successful tool evidence.

Why

The previous extractor sampled a short transcript tail and appended a small number of facts to auto.md. In practice, it was easy to miss durable product, technology-stack, collaboration, and cross-repository context. It also lacked a reliable queue and a precise retrieval layer.

Memory v2 separates the two concerns:

clean root turn
  -> bounded post-turn projection
  -> durable job
  -> background extraction
  -> transactional topic update + derived MEMORY.md

user query
  -> local exact/BM25F retrieval
  -> optional semantic topic selector
  -> bounded section attachment
  -> main model response

Implementation

Reliable post-turn capture

• Enqueues exactly once after a clean root loop stop.
• Skips sub-agents, aborted/error/filtered/length-truncated loops, greetings, and empty turns.
• Includes current-turn user input, final answer, bounded tool evidence, changed files, verification, and repository identity.
• Redacts secrets before the job is written and again before model extraction.
• Uses deterministic job IDs, atomic rename-based claiming, retry/backoff, stale lease recovery, bounded run history, and restart recovery.
• Keeps the foreground response independent from extractor latency; shutdown performs a bounded drain and leaves unfinished work on disk.

Markdown-first global store

• Uses one user-level namespace under userXcodeDir()/memory, shared across repositories.
• Treats topics/*.md as the only memory source of truth.
• Generates the compact, pinned core profile in MEMORY.md deterministically.
• Serializes writers with a cross-process lock and applies topic/MEMORY/index changes through recoverable transactions.
• Validates safe IDs and transaction paths to prevent traversal from damaged state files.
• Physically removes superseded or explicitly forgotten facts from all active recall paths.
• Preserves unrelated manual Markdown formatting and never watches or periodically scans topic files.

Hybrid recall

• Builds an in-memory fact/section index from valid topics.
• Runs exact ID/title/alias/path matching and BM25F locally without model tokens.
• Applies repository, type, freshness, relationship, and route-support signals before selecting candidates.
• Calls the semantic selector only when a historical query has plausible but ambiguous candidates; the selector receives a compact manifest, never all topic bodies.
• Packs only relevant sections into bounded dynamic attachments instead of mutating the stable system-prompt prefix every turn.
• Supports one late-bound recall pass when successful tool results reveal a new path, package, error code, or identifier.
• Adds the read-only memorySearch tool for explicit, scoped history lookup.

CLI and operations

• Initializes one root MemoryService and keeps sub-agents isolated from memory writing/search.
• Adds /memory, /memory status, /memory search, /memory explain, and /memory reload flows.
• Reports remembered, updated, forgotten, no-op, warning, and failed outcomes.
• Reloads manual topic edits only on explicit /memory reload or CLI restart; invalid topics are isolated instead of served from stale in-memory content.
• Documents the v2 storage model, recall behavior, configuration, privacy behavior, and legacy-file policy in Chinese and English.

Compatibility and legacy data

• There is intentionally no migration path.
• Existing user/project memory/auto.md files are ignored: they are not read, validated, moved, rewritten, or deleted.
• The legacy AutoMemory, getAutoMemory, and initMemories runtime APIs are removed and the public export snapshot is updated.
• Memory can be disabled with memory.enabled if an operational rollback is needed.

Correctness and safety coverage

Automated tests cover:

• root-only post-turn enqueue and incremental projections;
• job enqueue/claim/retry/recovery and stable identifiers;
• secret redaction and structured extractor output;
• topic parsing, manual-content preservation, invalid-topic isolation, and deterministic indexes;
• conflict replacement and complete removal of the old value;
• transaction serialization, idempotent generation recovery, and path validation;
• exact/BM25F retrieval, repository weighting, selector fallback, token packing, and recall state;
• attachment persistence/tombstones, late recall, and scoped memorySearch behavior;
• fresh v2 initialization while a legacy auto.md remains untouched.

Validation

• pnpm run ci
  • typecheck passed
  • lint passed
  • 93 test files passed
  • 938 tests passed, 3 skipped
  • core and CLI builds passed
• git diff --check passed
• Prettier checks passed for the design and user-facing documentation

Manual verification checklist

• Confirm first-run v2 layout and queue diagnostics.
• Ask x-code to remember a product, repository, stack, and working preference; verify topic and core-profile output.
• Verify exact, semantic, and cross-repository recall in a fresh session.
• Replace an existing fact and confirm the old value is absent from topics, MEMORY.md, search, and recall.
• Edit a topic manually, verify the current session does not watch it, then validate /memory reload and restart behavior.
• Corrupt one topic and confirm it is isolated and reported without serving stale content.
• Explicitly forget a fact and verify physical deletion.
• Exit with pending work and confirm the durable queue resumes on the next start.