docs: AI memory worker design + reorganize docs/ to canonical layout

[hanwencheng]

Summary

Two related changes in one PR. They land together because the docs reorganization establishes the canonical docs/plan/ location that the new memory plan inhabits.

1. New AI memory worker design plan + research

• docs/research/ai-memory-systems-survey.md — survey of 10 systems (Mem0, Letta/MemGPT, Zep/Graphiti, A-MEM, Cognee, MemMachine, LangMem, Claude memory tool, ChatGPT memory, OpenMemory MCP). Covers the four-type memory taxonomy (episodic / semantic / procedural / profile), three-stage pipeline shape (extract → consolidate → retrieve), storage substrates (vector / graph / JSON / files), retrieval mechanics (tool-call vs pre-call RAG vs full-context injection), portability formats (Letta .af, JSON Agents PAM, JSONL), privacy patterns, and the LoCoMo / LongMemEval benchmarks.

• docs/plan/agentkeys-memory-design.md — AgentKeys-specific design for evolving crates/agentkeys-worker-memory from today's blob-storage primitive (memory_put / memory_get) into a structured-memory service. Key invariants:

  • Worker never calls an LLM — embeddings come from caller; extraction lives client-side.
  • LLM never sees the whole memory — /v1/memory/search returns top-K snippets only.
  • LLM is replaceable without re-keying — memory format is LLM-agnostic JSONL on S3.
  • Per-actor + per-data-class isolation invariants from arch.md §17.5 preserved.

• Storage: one S3 object per JSONL line (bots/<actor>/memory/episodic/<YYYY-MM-DD>/<ulid>.enc) — atomic per-key PUT + cheap HEAD-for-dedup + clean K3 rotation. Brute-force cosine over a packed-binary index file on S3 is the v0 default (~50ms for 100K vectors at 1536 dim); vector DB is a deferred operator-elected cache layer.

• Prerequisite (M-1): envelope v3 lands in agentkeys-worker-creds as a separate PR before any memory-worker code change. Widens AAD to (operator_omni, actor_omni, service, k3_epoch) and adds an explicit K3 epoch byte to the envelope header.

• Plan has been through /plan-eng-review — 18 findings folded back in, 8 new test files spec'd, 4 critical failure modes covered.

2. Reorganize docs/ to the canonical layout

Matches the layout enforced by the agentkeys-docs skill:

docs/
├── arch.md          ← single source of truth (renamed from docs/spec/architecture.md)
├── spec/            ← developer specs (14 .md)
│   └── plans/       ← legacy execution plans (9 .md)
├── plan/            ← agent-authored pre-implementation plans (1 .md)
├── research/        ← third-party tech writeups (5 .md)
│   └── aiosandbox/  ← moved from docs/spec/aiosandbox/
├── wiki/            ← operator/integrator docs, auto-mirrored to GitHub Wiki (21 .md)
└── archived/        ← superseded/retired (25 .md)

Moves (29 total):

• `docs/spec/architecture.md` → `docs/arch.md` (single source of truth name)
• Repo-root `wiki/` (17 pages) → `docs/wiki/`
• `docs/spec/aiosandbox/` → `docs/research/aiosandbox/`
• 4 active operator runbooks: `cloud-setup.md`, `dev-setup.md`, `runbook-k3-rotation.md`, `v2-stage2-heima-deploy-and-test.md` → `docs/wiki/`
• 5 stale stage-N files archived: `operator-runbook-stage7.md`, `stage7-demo-and-verification.md`, `stage8-wip.md`, `v2-stage1-iteration-log.md`, `v2-stage1-migration-and-demo.md` → `docs/archived/`

Cross-link rewrites (80+):

• All 23 source files (Rust + scripts + CI workflows) referencing `docs/spec/architecture.md` updated.
• All 30+ files referencing repo-root `wiki/` updated, including per-doc relative-path depth shifts.
• `CLAUDE.md` references updated.
• `.github/workflows/publish-wiki.yml` mirror path updated to `docs/wiki/` in the same commit — wiki publishing stays green on next push to main.

Zero reorg-caused broken intra-repo links remain. (25 pre-existing broken refs to external content like `~/Projects/lifeKnowledge/` are flagged as out-of-scope TODOs.)

Test plan

• All file moves landed via `git mv` (jj can't colocate in a git worktree per CLAUDE.md `/create-pr` policy).
• `docs/arch.md` reachable from every internal link that previously pointed at `docs/spec/architecture.md`.
• `docs/wiki/` contains 21 .md files (17 original + 4 moved operator runbooks).
• `docs/archived/` contains 25 .md files (20 prior + 5 newly archived stage-N files).
• `.github/workflows/publish-wiki.yml` references `docs/wiki/` in both the `paths:` filter and `with: path:` step.
• Python-based link audit run repo-wide: 0 reorg-caused broken intra-repo links.
• Confirm `cargo build` is green after this PR lands (the moves touched Rust doc-comments that reference arch.md — these are comment-only, no compile impact, but worth verifying in CI).
• Confirm the next push to `main` mirrors `docs/wiki/` to the GitHub Wiki correctly.

Notes for reviewers

• The PR is large by file count (89 files) but most of the change is mechanical (renames + path updates). The substantive content is the two new docs in `docs/research/` and `docs/plan/`.
• Per the `/create-pr` policy in CLAUDE.md, no `Co-Authored-By` line — the commit author identity is the agent runtime that executed it.
• Outstanding TODOs (out of scope for this PR but flagged in the reorg report):
  • `docs/wiki/agent-role-and-usage-hdkd-per-agent-omni.md` is referenced by CLAUDE.md + arch.md but doesn't exist. Pre-existing gap from the wiki-location-policy migration that didn't fully complete.
  • 25 pre-existing broken refs to external content (`~/Projects/lifeKnowledge/`, non-existent crate paths, GitHub Wiki extensionless slugs that work on the published wiki).
  • arch.md is 2042 lines — within tolerance but borderline-bloated. New sections should consider extraction to `spec/.md` first.