Skip to content

Lesson Validation Methodology

Jump to bottom
Chris Sweet edited this page Jun 12, 2026 · 4 revisions

type: synthesis up: "Home_llm-wiki-memory-template" supports: "Three-Operations" tags: [lesson, testing, methodology]

Lesson: Validation Methodology

Template changes need two distinct kinds of validation: structural (does the template still instantiate cleanly?) and behavioral (does a live agent actually follow the new procedure?). PR #1 shipped initially with neither; the review caught six issues, four of which would have been caught by a structural test and one by a behavioral test.

Structural validation

Build a throwaway project from the branch, run instantiate.sh end to end, and verify with a small check script that:

  • All expected files exist at their expected paths
  • The generated CLAUDE.md and SCHEMA_<repo>.md contain the expected sections
  • The per-overlay surfaces install correctly
  • Any optional features (hooks, KG pipeline) install correctly when their flags are passed
  • Per-overlay setup-script flags work as documented

PR #4's self-test was 21 checks; PR #1's follow-up structural test caught the missing CLAUDE.md.template update (Priscila's point 6).

Behavioral validation

Open a Claude Code session in a freshly instantiated project and exercise the procedure as a user would:

  • Run /wiki-source on a test document. Observe whether the agent reads git config user.name rather than guessing it.
  • Verify the log entry's first bullet has the correct by: attribution.
  • Verify the wiki repo receives the expected number of commits (two per ingest: pages+index, then log).
  • Run an action that triggers the PostToolUse-Hook. Verify the hook fires, the agent receives the reminder, and the action proceeds (advisory, not blocking).

PR #1's point 7, the prompt-vs-command hook type bug, was only surfaced by behavioral testing. The hook installed correctly (structural pass) but broke the ingest flow when actually invoked (behavioral fail). See Lesson-Hook-Type-Matters.

The combination

Structural validation is fast and cheap; behavioral validation is slow but catches the cases structural cannot. For template changes, both are needed. Roughly:

  • Structural: a single check script, runs in seconds, repeatable, ~21 checks for the current template surface.
  • Behavioral: 10 to 15 minutes of manual interaction per overlay; hard to script because the agent's free-form decisions are part of what is being tested.

The current template has structural validation for the Claude Code overlay; Cursor and none overlays are structurally validated but not behaviorally, see Limitation-Untested-Overlays. Closing that gap requires running the same behavioral protocol against each overlay in a real session of the corresponding agent.

What to add to a PR description

For template changes, a PR description should state what kind of validation was performed. "Structural: 21/21 PASS. Behavioral: ran /wiki-source in live Claude Code session, 6/6 properties verified." Without that, reviewers either re-run the validation themselves or merge on trust.

See also

Clone this wiki locally