Native narrative memory for Agent Kaizen — a memory palace hypothesis

[LevyBytes]

Native narrative memory for Agent Kaizen — a memory palace hypothesis

So I was watching Fifth Element randomly with my family the other day, watched it countless times... it's one of the few sci-fi movies with really good replay value imo, and remembered the news reports I heard about Milla's mem palace project. And thought maybe that ancient memory palace concept could improve the data plane aspect of my Kaizen project? Well I dived in, and went back and forth with the best models I have access to today for review of the project concepts, mainly ChatGPT 5.5 pro extended (I forgot 5.6 Sol Ultra was made available today) and Claude Fable 5. I had those two duke it out over the details for a while and they made what seems like a decent step by step improvement plan by adopting some of those concepts (not code). And then folded that again between the two to write this discussion article. So, potentially useful slop begins below, let's fing go!

---

Everything below this line and including this line is LLM-generated and fact-checked against the actual codebases (a ChatGPT Pro deep-research assessment, adversarially reviewed and re-planned by Claude Fable 5, then edited down into this post). Machine homework — ponder accordingly.

Status: hypothesis. Nothing below is implemented. This is the reasoning, written down and posted for discussion before any code lands, because everything starts as an idea and the cheap time to break one is now.

Borrowing Leeloo's multipass for the concepts

The memory palace is ancient tech. Greek orators memorized entire speeches by walking through an imaginary building and leaving each idea in a specific room — the method of loci. It's a genuinely great model of memory: spatial, navigable, and verbatim. You don't remember a summary of the milk; you remember the milk flooding the hallway.

Earlier this year, Milla Jovovich — yes, Leeloo herself — and developer Ben Sigman turned that idea into an open-source AI memory system called MemPalace: wings for projects, rooms for topics, drawers holding the verbatim originals, per-agent diaries, save-before-compaction hooks, and a bounded wake-up context for session starts. Whatever anyone thinks of any single implementation detail, they put a really useful set of concepts into the open — and several of them map surprisingly well onto problems Agent Kaizen has been circling. So: genuine thanks and credit to both of them. An actress I've enjoyed since The Fifth Element shipping an AI memory tool is exactly the kind of plot twist 2026 keeps delivering, and I'm here for it.

To be clear about what this discussion proposes: Kaizen isn't taking MemPalace as a dependency and isn't lifting its code. The plan is concepts-only — translate the ideas into native Kaizen primitives, written in Kaizen's own methods. Ideas in, our own implementation out. The whole point of a project named after continual improvement.

The gap: narrative memory falls between transcript and proof

Agent Kaizen already does the part of memory well. Tasks, plans, verifications, evidence, evals, traces, and agent runs are written through deterministic commands into structured local records, so the next session restarts from evidence instead of vibes. That foundation isn't the problem.

The problem is the softer layer that real agent work runs on: "I noticed this but haven't verified it yet." "This may be flaky — need another run." "We chose option B because A broke on Windows." "I was about to inspect this when compaction hit." "A sub-agent found something suspicious that isn't a formal GOTCHA yet." That layer sits below verified records but above disposable scratch. It's not authoritative enough to become a proof or a LEARNED record, but it's too valuable to lose every time a session compacts, resets, or hands off between agent surfaces. Today it lives in the chat transcript, which means today it dies with the chat transcript.

The proposal is a native narrative-memory lane inside Kaizen's own data plane — not bolting on a second memory product. And the design has exactly one non-negotiable rule:

Memory advises. Kaizen records decide.

Memory can suggest what to inspect, remind an agent what it noticed, help draft a GOTCHA candidate. It can never rewrite truth, because a vague recollection silently competing with a verified conclusion would break the reason Kaizen exists.

flowchart TD
    RAW["Chat transcript / scratch<br/>(dies at compaction, reset, handoff)"] -->|"worth keeping, not yet truth"| NAR["Narrative memory lane<br/>observations · suspicions · follow-ups"]
    NAR -->|"review + promotion (existing path)"| CAND["Candidates<br/>GOTCHA · LEARNING · task updates"]
    CAND -->|"verified / accepted"| AUTH["Authoritative Kaizen records<br/>proofs · conclusions · LEARNED"]
    AUTH --> R0["R0 bounded session digest"]
    NAR -.->|"advisory only — never back-edits truth"| AUTH

Loading

What's being borrowed, concept by concept

Not the whole palace ontology — just the portable workflow concepts, each landing on a primitive Kaizen already has. The agent diary becomes a note event kind on the existing T5/T6 run ledger. Checkpoint-before-compaction becomes a documented session-notes workflow through the evidence plane (E1 → E3 → E4). The layered memory stack turns out to be surfaces Kaizen already ships, just named as layers. Wing-scoped recall becomes, if ever needed, a second native Kaizen instance acting as a shared local palace. Temporal facts get deferred entirely until real usage proves the existing revision history can't answer the question.

Layer	Kaizen surface	Role
Startup state	R0	Bounded, authoritative digest
Policy context	X5	Private policy / session rules
Narrative continuity	T6 note + optional R12	Advisory observations, session continuity
Deep recall	E4	Semantic / hybrid search over ingested evidence
Promoted truth	G*, L*, Q*, W*, ledger	Reviewed, authoritative records

Phase 1, move one: a native diary as a note event (~2 lines of code)

The T6 agent-event ledger is already the right home — append-only, redaction-checked on write, content-hashed, idempotent on replay. The only thing missing is a sanctioned non-blocking event kind for "I noticed this," and there's an exact precedent in the registry ("verification": ["point"] is already a non-span point kind):

# kaizen_components/schemas/registry.py — AGENT_EVENT_KIND_MARKERS
"verification": ["point"],
"note": ["point"],  # free-text observation / diary; non-span, never blocks completion

Plus "note" added to the event_kind enum in the same registry. Usage:

python kaizen.py T6 --agent-run-id AR_ID --payload-json \
  "{\"event_kind\":\"note\",\"marker\":\"point\",\"summary\":\"Suspect flaky CI on Windows runners\",\"body\":\"Two timeouts on test_backends_live; not reproduced locally.\"}" --json

That's a native agent diary with no new subsystem: local, redacted, hashed, replay-safe, attached to a real run, visible to reports — and inert for completion gating, because a note should preserve continuity, never block finalization.

Phase 1, move two: session notes through the evidence plane (zero code)

At task close and before compaction, the agent writes a short notes file and runs it through the pipeline that already has semantic search:

python kaizen.py E1 --file AI/work/notes/2026-07-10-session.md --task-id TASK_ID \
  --summary "Session notes: chunker comparison + open questions" --json
python kaizen.py E3 --id DOC_ID --json
# next session:
python kaizen.py E4 --query "why did we pick the recursive chunker" --semantic --json

That converts session-boundary memory from "chat that might disappear" into searchable local evidence, using a save point that's explicit and a recall path that already exists. It's the cleanest native equivalent of MemPalace's checkpoint-before-compaction hook — a documentation change, not a code change.

flowchart TD
    WORK["Agent session"] --> NOTE["T6 note events<br/>redacted · hashed · idempotent · non-blocking"]
    WORK --> FILE["Session notes at task close / pre-compaction<br/>AI/work/notes/&lt;date&gt;-session.md"]
    FILE --> E1["E1 ingest"] --> E3["E3 chunk"] --> E4["E4 semantic recall<br/>(next session)"]
    NOTE --> RUNS["Run ledger + R0 orchestration sections"]
    E4 --> R12["R12 memory-recall — optional, gated<br/>advisory_only: true"]

Loading

R0 stays sacred; R12 is earned

R0 is one of the most important contracts in Kaizen — the compact, curated "start here" payload every agent host trusts. It stays exactly as it is; no advisory memory gets injected into it, ever.

If notes and session-note ingestion turn out not to be enough — meaning agents keep re-asking already-answered questions often enough that it gets recorded as a GOTCHA — then and only then does a new read-only report op get built: R12 memory-recall, returning recent notes plus note-origin evidence hits with "advisory_only": true stamped on the payload, a normal usage denial when --query is missing, and SKIPPED (never fatal) when an optional recall path is unavailable. R12 isn't built because it sounds useful. It's built if the measurement says the gap survived Phase 1.

Phase 2 hypothesis: cross-project recall as a shared local Kaizen palace

The one thing a per-repo AI/db/kaizen.db genuinely cannot answer is cross-project narrative recall — "didn't I hit this exact auth weirdness in the other repo?" That's where MemPalace's wing concept earns its keep. But Kaizen doesn't need a foreign runtime to implement it, because Kaizen already supports alternate data-plane roots. A shared palace is just another Kaizen instance:

KAIZEN_REPO_ROOT=<dev-root>/kaizen-palace python kaizen.py K1 --json

<dev-root>/kaizen-palace/
  AI/db/kaizen.db              # its own native data plane
  notes/
    agent-kaizen/...           # one subdirectory per repo = the wing
    other-repo/...

The wing rides in origin_ref for free — E1 records the source path, so notes/agent-kaizen/2026-07-10.redacted.md self-identifies its project with zero new columns. At session boundaries, after all local writes commit, a redacted copy of the note gets ingested into the palace with the same E1/E3 commands under the palace root; reads come back through R12 --with-palace as a subprocess-to-self E4 query with a five-second timeout that fails open to SKIPPED. Unset the env var and the feature ceases to exist. No new dependency, no hosted service, no network, no new tables, no second product lifecycle — a deliberately shared local Kaizen data plane and nothing more.

flowchart TD
    A["repo A: session notes"] -->|"redact, then copy"| P["kaizen-palace/ — a second native Kaizen instance<br/>(KAIZEN_REPO_ROOT)"]
    B["repo B: session notes"] -->|"redact, then copy"| P
    P --> ING["palace E1 → E3<br/>wing = notes/&lt;repo-slug&gt;/ path prefix"]
    ING --> Q["R12 --with-palace<br/>subprocess-to-self E4 · 5s timeout · fail-open SKIPPED"]

Loading

The palace path is held to stricter rules than in-repo notes, because it widens the recall surface: no redaction pass, no export; no successful local commit, no palace handoff; and no palace hit ever rewrites authoritative state on its own — anything worth keeping goes back through the normal GOTCHA/LEARNING promotion path like everything else.

What's deliberately not being adopted

No MemPalace sidecar, import, or subprocess. No hosted memory or network transport of any kind. No graph-navigation layer, hallways, tunnels, or closet mirror — R0 and L10 already are the compact pointer layer. No contradiction detector. And no temporal facts table unless real GOTCHAs prove the existing revisions and ledger can't answer an "as-of" question; even then, the remedy would be one plain SQL table under a controlled predicate vocabulary, not a graph engine.

The discipline matters more than any single feature. A memory system's failure mode is becoming a second, fuzzier database that competes with the real one — and that would be a regression dressed up as progress. Kaizen should absorb the concepts that strengthen its existing data plane and reject the parts that add surface area without evidence.

How this hypothesis gets validated

Measured, not vibe-driven. Phase 1 gets instrumented with machinery that already exists: T6 note counts, how often later sessions actually hit note-origin evidence (visible through trace_events), a small eval set scored through eval_scores ("did the new session recall decision X without re-asking?"), and before/after bench_kaizen.py runs to prove no startup or write-path regression. Every expansion has a written trigger: R12 only if the re-asking GOTCHA recurs; the palace only if the recurring miss is specifically cross-project; filters on note documents only after pollution of normal evidence queries is recorded as a GOTCHA first. If T6 note plus session-note ingestion solves the problem, this stops at Phase 1 — a clean stopping point is a feature.

Open questions for this discussion

The first step is intentionally tiny: two registry lines for note events, plus documenting the session-notes workflow. The real question is where the line gets drawn after that. Should Kaizen stop at per-repo narrative continuity unless cross-project recall becomes a proven recurring miss? Or is the shared local palace worth designing early and leaving disabled until the eval gate trips? Current bias: Phase 1 first, measurement second, palace only after evidence. Poke holes below — that's what this post is for.

References

• MemPalace repository — Milla Jovovich & Ben Sigman
• MemPalace — the palace concept
• MemPalace — knowledge graph
• MemPalace — auto-save hooks
• Method of loci — the original memory palace
• Agent Kaizen