AgentsMdInversion
title: AGENTS.md Inversion type: technique created: 2026-05-28 last_updated: 2026-05-28 related: ["radar/techniques/BuildingSkillsForClaude", "radar/techniques/PersonalLLMCodingBenchmark", "radar/techniques/LLMEvaluationMethodology", "radar/tools/SkillOpt"] sources: ["https://www.stet.sh/blog/how-i-used-codex-to-improve-its-own-agents-md"] radar_quadrant: Techniques radar_ring: Assess radar_position: center
AGENTS.md Inversion is a documented failure mode in which iteratively self-improving an agent instruction file (AGENTS.md, CLAUDE.md, or equivalent) raises average benchmark scores while silently degrading performance on specific task types. Named and demonstrated by the author of stet.sh in a 2026 experiment using Codex to edit its own AGENTS.md across 8 candidate versions.
| Element | Detail |
|---|---|
| Subject | Codex (gpt-5.5, medium reasoning) editing its own AGENTS.md |
| Grader | Separate model (gpt-5.4) scoring tests, equivalence to merged human PRs, code review quality, footprint risk, token usage, craft rubrics |
| Training slice | n=5 historical Stet PRs |
| Holdout | n=10 separate tasks, run only on the best candidate |
| Goal given to Codex | Improve benchmark performance by editing AGENTS.md |
The loop ran: hypothesis → test on real work → inspect failures → revise rule → run holdout → validate. Iteration 7 emerged as the best candidate, centred on an "obligation ledger" rule requiring the agent to identify obligations, owners, and validation paths before editing.
On the n=5 training slice, the best candidate looked promising: it fixed a missed task and improved several craft scores. On the clean n=10 holdout, it regressed:
- Correctness: -0.20
- Scope discipline: -0.49
- Footprint: widened
- Tokens and tool calls: climbed
Trace analysis showed the agent had improved at local implementation craft (clearer names, structured logs) but degraded at boundary judgment — narrowing broad requests, over-documenting, building parallel contracts instead of extending existing ones. Average metrics rose; specific task types silently regressed.
The author's term — "AGENTS.md inversion" — describes the pattern: optimization on a small training set produces rules that hyper-target observed failures while damaging behaviours that were already working.
The author argues AGENTS.md should be treated as "a tunable part of the harness" rather than a vibes-driven living document. Self-improvement loops require held-out validation to detect inversion; recursive self-editing in a void is not sufficient. Five questions proposed for editing shared instruction files:
- What behaviour should change?
- Which tasks expose it?
- Does the edit improve behaviour or vibes?
- What got worse?
- Did the holdout agree?
Caveats noted by the author: single repo, n=10 holdout — directional, not statistically significant.
Placed in Techniques / Assess / center.
Center position because the experiment is single-author with small samples and a soft pitch for the author's eval tool, but the failure mode it identifies is mechanistically plausible and matches anecdotal observations from teams maintaining shared CLAUDE.md / AGENTS.md files. The technique here is the discipline: never edit a shared instruction file without a held-out task set to detect regression. This sits as a counterpoint to radar/tools/SkillOpt, which solves the same problem (validation-gated skill editing) with a more rigorous mechanism. The two together suggest a consistent direction: instruction files are tunable harness components, and editing them without held-out gating is the practice to retire.
Distinct from radar/techniques/PersonalLLMCodingBenchmark (which advocates a personal benchmark for tool selection) — this technique applies the same evaluation discipline specifically to instruction-file edits, where the inversion failure mode is most acute because edits compound silently across sessions.
Trial gate: at least one AGENTS.md or CLAUDE.md edit accepted only after passing a held-out task set, with a documented case where a candidate edit improved training scores but failed the holdout and was rejected.
