-
Notifications
You must be signed in to change notification settings - Fork 2
Limitation Untested Overlays
type: synthesis up: "Agent-Overlays" criticizes: "Agent-Overlays" tags: [limitation, agents, testing]
The template ships overlays for claude-code, cursor, and none. Only claude-code is currently behaviorally validated. The others are structurally validated only: files install correctly, but no one has confirmed in a live session that the agent actually follows the procedures.
The overlay's files instantiate to the right paths. The setup.sh runs without error. Generated content has the expected sections. None of this guarantees that a real agent behaves the way the procedures intend.
A real agent session, in a fresh project, runs the named operations (ingest, query, lint) and produces:
- Correct attribution lines (
by: <human> via <agent>) - Two commits per ingest (pages+index, then log)
- Bidirectional cross-references
- A verification-gate pass before commit
- Honest reporting (no projections-as-fact)
For Claude Code, these have been seen in live sessions. For Cursor, the rules are written but unverified. For none, the procedures live only in CLAUDE.md and SCHEMA, and whether a non-instructed agent actually follows them is open.
The procedure-on-paper is the easy part. Whether each agent's surface actually causes the procedure to fire is empirical. PR #1 found that even within the validated overlay, a hook installed correctly (structural pass) but broke ingests behaviorally (behavioral fail), see Lesson-Hook-Type-Matters. The same kind of mismatch could exist in Cursor or none and not be visible until tried.
For each unvalidated overlay:
- Instantiate a fresh project with that overlay
- Run each of the three operations in a real session of the corresponding agent
- Verify the six properties listed above
- File any deviations as issues against the template
This is the behavioral half of Lesson-Validation-Methodology.
The template's README and per-overlay READMEs flag unvalidated overlays explicitly with phrases like "untested-live". This is honest reporting, not a hidden gap.