Skip to content

Lessons Learned From model_fusion

Jump to bottom
Chris Sweet edited this page Jun 12, 2026 · 3 revisions

type: synthesis up: "Home_llm-wiki-memory-template" related:

Lessons Learned From model_fusion

A field report from a long Claude Code session in the PaperAnalyticalDeviceND/model_fusion derived project that surfaced empirical gaps between what the LLM-Wiki pattern claims and what derived projects actually have installed. The session generated 12 improvement recommendations plus one separate finding (the "recursive bootstrap problem"). Verification against this template's HEAD (88f4624, 2026-06-12) found 0 of 12 fully addressed, 3 partially addressed, 9 still missing — and the recursive bootstrap problem RESOLVED in PR #25.

This is filed as a sister page to Applied-Example-Microelectronics-Tutor-Demo — a second empirical test of the template, from a different domain (NIR + PAD pharma analytics rather than undergraduate electronics) and a different scale (43 wiki pages, 24 experiments, ~5 months of active curation).

Headline: 0 of 12 original recommendations are fully addressed by the current template; 3 are partially addressed (#1 SessionStart, #6 PostToolUse compounding metric — the existing hook is where it would land, and #9 reciprocal edges, refined below now that PR #16 has shipped the inverse-edge vocabulary); 9 are still missing. The separate "recursive bootstrap problem" finding has been RESOLVED in PR #25 via the new scripts/instantiate.sh --dev-self flag. v1's recommendations hold up against the latest template.

Verification table (vs current template HEAD 88f4624, 2026-06-12)

# Recommendation Status
1 SessionStart auto-load of index_<wiki>.md + recent log PARTIAL — opt-in SessionStart hook scaffold exists; it emits a text reminder but does not load wiki content into context
2 UserPromptSubmit synthesis-question hook STILL MISSING
3 Stop hook for unfiled substantive content STILL MISSING
4 analysis and decision page types in SCHEMA STILL MISSING
5 primary_wiki + cross-wiki rules in CLAUDE.md STILL MISSING
6 PostToolUse hook records compounding-ratio metric STILL MISSING
7 Lint as a scheduled / forced operation STILL MISSING
8 Claude-memory vs wiki boundary declared in CLAUDE.md STILL MISSING
9 Force reciprocal-edge discipline at write time PARTIAL — inverse-edge vocabulary now shipped in wiki/Edge-Types.md.template via PR #16 (commit b86c9e4); not yet wired into wiki/init-wiki.sh's generated SCHEMA page-type list, and the PostToolUse hook does not enforce reciprocal edges. Verification-Gate prose still carries the check. See Limitation-Inverse-Edges-Missing (template issue #3)
10 Project-typed SCHEMA at instantiate time STILL MISSING
11 CLAUDE.md re-injection mid-session STILL MISSING
12 Visible filing-footer in chat transcript STILL MISSING

Two lessons the current template HAS internalized that travel with these recommendations:

  • Hook type matters (Lesson-Hook-Type-Matters + the explicit comment in posttooluse-hook.sh): advisory wording in prompt-type hooks does not work. Only command-type hooks that exit 0 can be advisory. Every hook recommendation below assumes command-type.
  • Parallel file drift (Lesson-Parallel-File-Drift): every prose procedure added to the system increases drift surface. New page types, new SCHEMA sections, new hook scripts must be paired with explicit DRY references or test coverage.

Partial recommendations (refined against current template)

#1 — SessionStart hook should auto-load index + recent log (not just remind)

Current state: the template ships an opt-in SessionStart hook (via setup.sh --hook). The script at wiki/agents/claude-code/templates/session-start-hook.sh emits a <system-reminder> block reading roughly: "This project uses the wiki at wiki/${REPO_NAME}.wiki/ as durable memory ... Update the wiki proactively." It tells the agent the wiki exists. It does not load the wiki's current state into context.

The gap: the agent operates with awareness that the wiki exists but no awareness of what's in it. When the user asks "tell me about X," the agent reads the wiki on demand. That's RAG-on-the-wiki, not memory.

Fix: extend the existing session-start-hook.sh to actually inject content. Append something like:

echo
echo "## Wiki current state — index"
echo
cat "wiki/${REPO_NAME}.wiki/index_${REPO_NAME}.md"
echo
echo "## Wiki current state — last 5 log entries"
echo
awk '/^## \[/{c++} c<=5' "wiki/${REPO_NAME}.wiki/log_${REPO_NAME}.md"

Cost: a few thousand tokens at every session start. Payoff: the wiki actually functions as memory. This is the central change that converts "wiki as queryable filesystem" into "wiki as compounded memory."

Default: should be on by default, not opt-in. Lesson learned this session: capabilities shipped opt-in are statistically off (two of three inspected projects had the existing SessionStart hook turned off).

#9 — Mechanical enforcement of reciprocal typed-edges

Current state: wiki/agents/verification-gate.md includes the criterion: "Every new typed-edge reference in frontmatter (extends:, supports:, criticizes:, up:) has the reciprocal edge on the target page." This is prose the agent is supposed to read and apply before committing. No mechanism checks. Additionally, Limitation-Inverse-Edges-Missing documents (as issue #3) that the inverse-edge vocabulary itself is unimplemented — so the agent falls back to generic related: for back-references, degrading the typed graph over time.

The gap: a prose checklist run only when the agent remembers; the vocabulary needed for clean reciprocal edges doesn't exist yet.

Fix: two parts.

(a) PR #16 (commit b86c9e4) shipped the inverse-edge vocabulary at wiki/Edge-Types.md.template: 12 forward predicates and their inverses (supportedBy, criticizedBy, hasPart, …) with the owl:inverseOf mapping the Knowledge-Graph-Pipeline (shipped in PR #14 as scripts/kg/) consumes. The remaining work is wiring the inverse names into wiki/init-wiki.sh's generated SCHEMA page-type frontmatter list so that derived projects' SCHEMA validates them, and into the per-overlay procedures' frontmatter check.

(b) Extend the existing opt-in PostToolUse hook (already command-type; already wired to Write|Edit on wiki paths) to parse the edited file's frontmatter, look up the target page, and check whether the reciprocal edge exists. If missing, inject:

"Page X declares extends: Y but Y has no extended_by: X. Add before commit, or justify the asymmetry."

The hook is shell so it can't follow body-text wikilinks easily, but frontmatter is YAML and the check is straightforward with yq or grep.

Still-missing recommendations

#2 — UserPromptSubmit command-type hook for synthesis-question detection

Gap: discipline gates fire on agent reasoning, not on user input. By the time the agent could evaluate filing, the answer is already produced and the conversation has moved on. Witnessed in the model_fusion session: the CRC auth architecture analysis, the compute-pool decision matrix, the install-script-vs-symlink-vs-plugin debate all got answered substantively and dropped.

Fix: a UserPromptSubmit command-type hook pattern-matches on synthesis verbs (why, compare, what's the best, should we, how does X relate to Y, is there a tradeoff between, tell me about) and injects:

"Synthesis question detected. Before answering: (a) Query index_<wiki>.md for an existing page on this topic. (b) If a page exists, your answer should build on it. (c) After answering, evaluate filing as an analysis or decision page. Discipline gate "holding findings until they're stronger" applies; default YES, justify NO."

Must be command-type per Lesson-Hook-Type-Matters. Default on.

#3 — Stop command-type hook for unfiled substantive content

Gap: even with the discipline gates and the UserPromptSubmit hook from #2, an agent can complete a turn having answered a substantive question and never reached the wiki. The conversation ends, the finding falls on the floor.

Fix: a Stop command-type hook scans the assistant's recent response for substantive content markers (multi-option tables, "comparison," "decision," "tradeoff," "we picked X because") and injects:

"Substantive analysis detected: [extracted summary]. Was this filed? If yes, name the page. If no, file before the next user prompt or justify why this is not wiki-worthy."

Tune detection conservatively to avoid false-positive fatigue.

#4 — analysis and decision page types in SCHEMA

Gap: current SCHEMA types are concept, entity, source-summary, synthesis, index, comparison, untyped. None fit "user asked a synthesis question; the answer is fileable" or "we made a design decision and rejected alternatives." Without destination types, the Discipline-Gates rationalization "holding findings until they're stronger" fires correctly but the agent has nowhere clean to file.

Fix: add two types to the template's SCHEMA.

  • analysis: an answer to a synthesis question that emerged from discussion (not from a planned experiment). Required sections: Question, Context, Analysis, Conclusion, Open follow-ups. Frontmatter must include derived_from: listing source pages synthesised.
  • decision: a design choice with rationale. Required sections: Question, Options considered (with pros/cons), Decision, Rejected alternatives, Revisit triggers. Frontmatter must include decided_at: and optionally superseded_by:.

Parallel-file-drift caveat (per Lesson-Parallel-File-Drift): these page types need to be declared in SCHEMA (both create-mode heredoc and update-mode append in init-wiki.sh), referenced from the verification gate's frontmatter check, and acknowledged in the discipline gates as filing destinations. Three places to keep in sync.

#5 — primary_wiki declaration + cross-wiki filing rule in CLAUDE.md

Gap: when a person has multiple llm-wiki projects and a session crosses topics (extremely common in long sessions), analytical content can fit two wikis. Without a rule, it gets filed to neither. Witnessed in the model_fusion session: content fit both AI_skills and model_fusion and landed in neither.

Fix: every project's CLAUDE.md (and the template's CLAUDE.md.template) should declare:

primary_wiki: <path>
cross_wiki_rule: |
  File project-local content to primary_wiki. For content useful to
  another wiki, file to primary_wiki with a `cross_wiki_link:`
  frontmatter field pointing at the canonical home, and notify the
  user that an upstream PR is warranted.

This eliminates the "where does this go?" ambiguity that causes content to be dropped entirely.

#6 — PostToolUse hook records compounding-ratio metric

Gap: the pattern's value claim is "single source touches 5-15 pages" (per Three-Operations). No direct measurement of whether this is actually happening. A given project might be doing real compounding or might be filing each source as one isolated page (file-mirror behaviour). No signal.

Fix: extend the existing PostToolUse command hook (already shipped opt-in via PostToolUse-Hook) to record, to a side log: (timestamp, page_modified, related_pages_modified_in_same_session, session_id). Compute compounding ratio = average pages touched per ingest-session.

Ratio ≈ 1 → wiki is being used as file-mirror documentation. Ratio > 3 → real compounding is happening.

Surface in /wiki-lint output. Trend down over weeks is a leading indicator of pattern decay.

#7 — Lint as a forcing schedule

Gap: Three-Operations defines Lint as a maintenance pass. No project in any inspected wiki has a trigger to actually run it. Maintenance debt accumulates silently — inline TODOs in pages, broken one-way edges, stale facts. model_fusion's recent log entries specifically mention deferred refresh items (e.g., "future Final-Report refresh should swap in this plot") that a lint pass would catch and schedule.

Fix: two parts.

(a) The SessionStart hook (#1) additionally checks wiki/<repo>.wiki/.lint_state.json for the last lint date. If > 7 days, inject:

"Wiki hasn't been linted in N days. After completing the current user request, run a lint pass before the session ends."

(b) Make the Lint operation's concrete checks more explicit in SCHEMA: pages untouched in 90 days, inline TODOs aggregated, one-way typed edges flagged (until inverse-edge vocab lands per #9), concepts mentioned in ≥3 pages without their own page flagged for promotion, external citations in synthesis pages without source-summary pages flagged.

#8 — Don't conflate Claude-memory and the wiki

Gap: Claude Code has its own per-user persistent memory system (~/.claude/projects/<slug>/memory/). The wiki is project-scoped and LLM-owned. The boundary is ad-hoc and currently misallocates content in both directions — user-preference facts end up in both, project decisions end up in neither.

Fix: CLAUDE.md.template should explicitly declare:

  • Claude-memory holds: user identity, preferences, workflow style, cross-project guidance. Persists across all sessions for this user.
  • Wiki holds: project-specific knowledge, synthesis, decisions, experiment results. Persists across all sessions for this project.

When a fact emerges, the Stop hook (#3) or the filing-confirmation footer (#12) prompts the agent to choose the right home. Defaults: user-shaped → Claude-memory, project-shaped → wiki.

#10 — Project-typed SCHEMA at instantiate time

Gap: the instantiate script seeds the same SCHEMA for every project type. A research wiki (model_fusion: experiment-centric) and a documentation wiki (AI_skills: skill-centric) have very different content shapes. The generic SCHEMA forces every project to invent its own conventions, which is exactly the parallel-file drift surface to avoid.

Fix: scripts/instantiate.sh asks for a project type and seeds SCHEMA with type-specific page types and lint rules.

  • Research: experiment first-class. Aggressive lint on experiment-to-synthesis linkage.
  • Documentation: skill / module / API first-class. Aggressive lint on per-page README parity.
  • Personal knowledge: theme / thread / journal. Aggressive lint on thread freshness.

The analysis and decision types from #4 are universal across all three.

#11 — CLAUDE.md re-injection mid-session

Gap: CLAUDE.md loads at session start. By turn 100+ of a long conversation (the model_fusion session that produced this report is a clear example), its instructions are buried under context and don't fire reliably. The wiki-discipline directives lose force as the session ages. The SessionStart hook from #1 re-injects state at session start but not during a session.

Fix: a periodic UserPromptSubmit command-type hook (every N turns, or on topic-shift detection) re-injects the wiki-discipline section of CLAUDE.md as a <system-reminder>. Topic-shift detection can be keyword-based: prompts containing wiki page names, file/save/ keep verbs, or synthesis questions trigger a re-prime.

#12 — Visible filing-footer in chat transcript

Gap: when the LLM does file a wiki page, the user has no signal beyond a Write/Edit tool call lost in scrollback. Filing is invisible. No feedback loop, no accountability.

Fix: the Stop hook (#3) or a specialised filing-confirmation hook appends to the assistant's final response:

"📝 Filed: Page-Name (new analysis page)" or "📝 Updated: Page-Name (added section on X)"

Makes filing decisions accountable, gives the user a chance to redirect, creates an audit trail in the chat transcript that mirrors the log. Trade-off: this is user-visible chrome; some sessions don't want it. Default on with a per-project "no-filing-footer" flag.

Implementation priority

If a project can do only three things, in order:

  1. #1 — Extend the SessionStart hook to actually load index + log into context (not just remind). Single highest-leverage change. Without it, the wiki is not memory; every other improvement is conditional on this one. Smaller delta from current template than it sounds — the hook scaffold is already shipped, only the content needs to change.

  2. #4 — Add analysis and decision page types to SCHEMA. Without destination types, the existing Discipline-Gates have nowhere clean to file query-driven content even when they fire correctly.

  3. #2 — UserPromptSubmit command-type hook for synthesis-question detection. Closes the loop on the input side; complements the existing discipline gates which only address the agent-reasoning side.

After those three, #9 (mechanical reciprocal-edge enforcement) is the next priority because it addresses Limitation-Inverse-Edges-Missing which the template authors have already prioritised.

The recursive bootstrap problem — RESOLVED (PR #25)

A separate finding from the same session, originally flagged here as a template UX bug: a fresh clone of crcresearch/llm-wiki-memory-template did not land a contributor with llm-wiki context. The template had CLAUDE.md.template (placeholders), no wiki/<repo>.wiki/, and no hooks installed. So when someone cloned the template to develop on the template itself, they lost the very mechanism they were trying to improve. The template authors maintain their llm-wiki at the GitHub Wiki level, but cloning the template repo did not pull it down.

Resolved in PR #25 (commit 88f4624, "instantiate.sh: add --dev-self flag for template-contributor self-dogfooding"). Running scripts/instantiate.sh --dev-self now does exactly the dev-mode bootstrap this finding proposed:

  1. Expects the template's own GitHub Wiki cloned to wiki/llm-wiki-memory-template.wiki/ (the contributor performs the wiki clone explicitly before running the flag).
  2. Renders a real (non-.template) CLAUDE.md targeting the template repo itself, with the agent pointed at the local wiki.
  3. Installs SessionStart and PostToolUse hooks into .claude/hooks/.
  4. Gitignores the dev-mode artifacts (CLAUDE.md, the wiki sub-tree, the installed hooks) so they never propagate into derived projects.

The underlying principle holds: the template should eat its own dogfood at the filesystem level, not just on GitHub Wiki. PR #25 made that explicit.

Methodology for verifying this report in your project

Don't trust either this report or the template's own wiki claims about what's installed. Verify against the filesystem:

  1. Check what's actually enforced: cat .claude/settings.json for hook declarations. The wiki tells you intent; settings.json tells you what fires.
  2. Check what's shipped as prose: ls wiki/agents/ and read each file. Verification gates and discipline gates are markdown the agent is told to consult, not enforcement.
  3. Check your template version: compare your wiki/init-wiki.sh line count and content against the template's git history with git log --format="%h %ai %s" -- wiki/init-wiki.sh from a template clone. If your version's diff against an older template commit is small, you missed everything added since then.

The gap between (1)-(2) and "what your project actually does" is the empirical implementation gap. It tends to be wider than wiki status pages acknowledge.

Provenance

Originally verified 2026-06-12 against fc2472a (2026-06-04, "Add Edge-Types vocabulary page + Variant 1 inline annotation conventions"), a fork-local SHA that does not resolve in crcresearch/llm-wiki-memory-template's current main. Refreshed same-day against:

  • crcresearch/llm-wiki-memory-template at HEAD 88f4624 (2026-06-12, "instantiate.sh: add --dev-self flag for template-contributor self-dogfooding").
  • Local checkout at /Users/csweet1/Documents/projects/CRS_research/llm-wiki-memory-template/.

The same-day refresh found three commits between the original verification and HEAD that touched the 12 recommendations: PR #14 (scripts/kg/, the KG pipeline), PR #16 (wiki/Edge-Types.md.template, inverse-edge vocabulary), and PR #25 (--dev-self, the recursive bootstrap resolution). Rows 9 and the recursive-bootstrap section updated accordingly; rows 2-8 and 10-12 unchanged. See Wiki-Corrections-Log for the receipt.

Three derived projects sampled for empirical state:

  • chrissweet/AI_skills (current template snapshot, no hooks installed).
  • PaperAnalyticalDeviceND/annotated_chemopad (SessionStart and PostToolUse hooks both installed).
  • PaperAnalyticalDeviceND/model_fusion (instantiated from initial skeleton commit 59ddbe1 on 2026-05-13; missing every substantive feature added since).

The 12 recommendations were drafted from observation about how the wiki failed to function as memory in a long general-purpose session, then verified against the current template's actual files (not its wiki claims) for this report.

See also

Clone this wiki locally