Skip to content

Limitation Untested Overlays

Jump to bottom
Chris Sweet edited this page May 31, 2026 · 2 revisions

type: synthesis up: "Agent-Overlays" criticizes: "Agent-Overlays" tags: [limitation, agents, testing]

Limitation: Untested Overlays

The template ships two overlays, claude-code and cursor, plus a minimal mode invoked by --agent=none in scripts/instantiate.sh that installs no overlay at all (there is no wiki/agents/none/ directory; minimal mode ships only the llm-wiki core). Only claude-code is currently behaviorally validated. cursor is structurally validated only: files install correctly, but no one has confirmed in a live session that the agent actually follows the procedures. Minimal mode is by definition not validated against any specific agent.

What "structurally validated" means

The overlay's files instantiate to the right paths. The setup.sh runs without error. Generated content has the expected sections. None of this guarantees that a real agent behaves the way the procedures intend.

What "behaviorally validated" means

A real agent session, in a fresh project, runs the named operations (ingest, query, lint) and produces:

  • Correct attribution lines (by: <human> via <agent>)
  • Two commits per ingest (pages+index, then log)
  • Bidirectional cross-references
  • A verification-gate pass before commit
  • Honest reporting (no projections-as-fact)

For Claude Code, these have been seen in live sessions. For Cursor, the rules are written but unverified. For minimal mode (--agent=none), no agent files are installed at all: the procedures live only in CLAUDE.md and SCHEMA, and whether a non-instructed agent (OpenCode, Pi, or any other) actually follows them by reading those is empirical and currently open.

Why it matters

The procedure-on-paper is the easy part. Whether each agent's surface actually causes the procedure to fire is empirical. PR #1 found that even within the validated overlay, a hook installed correctly (structural pass) but broke ingests behaviorally (behavioral fail), see Lesson-Hook-Type-Matters. The same kind of mismatch could exist in Cursor or none and not be visible until tried.

How to address

For Cursor:

  1. Instantiate a fresh project with --agent=cursor
  2. Run each of the three operations in a real Cursor session
  3. Verify the six properties listed above
  4. File any deviations as issues against the template

For minimal mode there is no fixed target: the test depends on which non-instructed agent is hypothesised to follow CLAUDE.md and SCHEMA. Run the same protocol against whichever agent is being evaluated, and file results as issues describing the agent and version.

This is the behavioral half of Lesson-Validation-Methodology.

Documented status

The template's README and wiki/agents/README.md flag unvalidated paths explicitly with phrases like "Shipped but not yet validated" (for Cursor) and "Likewise unvalidated" (for minimal mode). This is honest reporting, not a hidden gap.

See also

Clone this wiki locally