Semantic Memory Evaluation Runbook

Semantic Memory Evaluation Runbook

This runbook preserves the operating procedure from issue #362 after the implementation epic closed. Use it when memory behavior changes enough that the project needs a fresh measurement, or when a production symptom suggests memory quality may have regressed.

When to Rerun

Run the semantic memory evaluation after changes that affect:

• memory extraction prompts or extraction filtering
• fact or episode storage metadata
• retrieval scoring, thresholds, overfetch, or role weighting
• prompt context assembly
• scoped memory or reflection behavior
• the active memory store growing enough that retrieval behavior may have shifted

Also run it when users report missed context, irrelevant retrieved facts, repeated preference corrections, or other memory-related friction.

Inputs

The evaluation uses local operator data. Do not commit probe files, output JSON, markdown summaries, or copied memory stores unless they are explicitly redacted fixtures.

Typical private inputs:

• DATA_DIR/memory/<chat_id>/
• DATA_DIR/history/<chat_id>/
• a private Layer 1 or Layer 2 probe file, usually probes.jsonl

The probe JSONL schema is documented by the loaders in src/kai/eval/retrieval_scoped.py and src/kai/eval/behavioral.py. Lines beginning with # are comments.

Snapshot Discipline

Prefer running against a copied data directory instead of the live store. The copy should include both memory/ and history/ when running Layer 2 with pollution probes or Layer 3 friction analysis.

Example shape:

export KAI_DATA_DIR=/tmp/kai-memory-eval-data
mkdir -p "$KAI_DATA_DIR"
cp -a /path/to/data/memory "$KAI_DATA_DIR/"
cp -a /path/to/data/history "$KAI_DATA_DIR/"

Run the commands from a Kai checkout with the virtual environment active, or call .venv/bin/python directly.

Layer 1: Retrieval-Only

Layer 1 answers whether retrieval returns the expected stored fact for each probe. It reports precision@K, recall@K, MRR, fraction-in-prompt, latency, and drift count.

.venv/bin/python -m kai.eval.retrieval_scoped <chat_id> \
  --probes /path/to/probes.jsonl \
  --output /tmp/memory-layer1.json

Use sweep mode when tuning retrieval parameters:

.venv/bin/python -m kai.eval.retrieval_scoped <chat_id> \
  --probes /path/to/probes.jsonl \
  --sweep \
  --output /tmp/memory-layer1-sweep.json

Read drift_count first. A high drift count means the probe set no longer matches the store, so refresh probes before interpreting retrieval quality.

The scoped harness takes further flags when you need them: --include-details (per-probe output), --projects and --non-project-only (restrict which memory-project scopes run), --floor (relevance threshold override), --user-weight / --assistant-weight / --episode-summary-weight (re-rank weight overrides), and --overfetch. Because retrieval is scope-aware, the report also covers cross-scope exclusion safety: whether rows from inactive projects stayed excluded, which matters as much as the hit metrics.

Layer 2: Behavioral A/B

Layer 2 answers whether retrieved memory improves generated answers compared with the same prompts without memory. It runs paired memory-on and memory-off generations and scores them with a judge model.

.venv/bin/python -m kai.eval.behavioral \
  --probes /path/to/probes.jsonl \
  --user-id <chat_id> \
  --output /tmp/memory-layer2.json

Use --pollution-lines N only when testing prompt-noise sensitivity. The default 0 is the normal rerun.

Useful signals:

• memory_wins: memory is improving answers.
• memory_loses: memory is actively hurting answers and should trigger investigation.
• both_wrong: usually means retrieval missed or the probe is outside what memory can answer.
• ties: memory may be irrelevant for those probes.

Layer 2 costs model calls. Keep probe count intentional.

Layer 3: Longitudinal Friction

Layer 3 scans chat history for memory-related friction signals and buckets them around the milestone boundaries from #362.

.venv/bin/python -m kai.eval.friction \
  --user-id <chat_id> \
  --output /tmp/memory-layer3-friction.json \
  --markdown-summary /tmp/memory-layer3-friction.md

For a quick sanity check:

.venv/bin/python -m kai.eval.friction \
  --user-id <chat_id> \
  --sample-days 14 \
  --output /tmp/memory-layer3-friction-sample.json \
  --markdown-summary /tmp/memory-layer3-friction-sample.md

Do not use --include-snippets for public reporting. The markdown summary omits surface text by design.

Additional harnesses

Beyond the three layers, src/kai/eval/ carries purpose-built harnesses worth knowing:

• modeswitch - the acceptance procedure for flipping MEMORY_ENABLED, proving both modes behave before and after a toggle
• memory_backend_gate - checks a backend's eligibility for memory extraction duty
• replay - re-runs recorded exchanges against the current pipeline
• gen_collision_probes and probe_corpus_check - generate and validate collision-probe corpora for canonicalization testing
• extraction - the extraction-quality harness

Each is a python -m kai.eval.<name> entry point with its own --help.

Reporting

Post a GitHub issue only when a rerun produces an actionable result:

• a regression against the last comparable baseline
• a high drift count requiring probe refresh
• memory-loses cases that expose a concrete failure mode
• a friction trend that contradicts expected behavior
• a methodology gap in the harness itself

A normal healthy rerun does not need a new issue. Store the private output locally and summarize only the non-sensitive aggregate numbers if public reporting is useful.

When reporting publicly, include:

• run date
• code revision or release
• probe set identity or hash, not private probe text
• data snapshot date
• Layer 1 metrics and drift count
• Layer 2 win, loss, tie, both-wrong counts
• Layer 3 trend summary, if run
• concrete follow-up issue links for any actionable findings

Historical Anchor

Issue #362 is the original semantic memory evaluation epic. It contains the design rationale and initial Layer 1 and Layer 2 result comments. The implementation sub-issues were:

• #363: structured retrieval logging
• #365: Layer 1 retrieval evaluation
• #369: Layer 2 behavioral A/B evaluation
• #373: Layer 3 longitudinal friction analysis

Keep #362 as historical context, not as the live tracking surface.