SelfImprovingAgentHarness
title: Self-Improving Agent Harness type: technique created: 2026-06-22 last_updated: 2026-06-22 related: ["radar/techniques/BuildingEffectiveAgents", "radar/techniques/ScalingLongRunningAgents", "radar/techniques/MaintainabilitySensorsForCodingAgents", "radar/techniques/YearBuildingWithLLMs", "radar/techniques/AgentsMdInversion"] sources: ["https://www.henrypan.com/blog/2026-05-25-self-improvement-harness/", "https://github.com/workofart/harness-experiment"] radar_quadrant: Techniques radar_ring: Assess radar_position: center
Self-Improving Agent Harness is the technique of having an AI agent autonomously modify the harness around a model: the scaffolding that controls what the model sees, which tools it can call, and how environment feedback is delivered. Henry Pan documented a six-week experiment running roughly 1,288 self-improvement loop iterations, 1,226 experiment runs, ~248,824 task steps, and ~881M input tokens against Terminal Bench 2.0, across GPT-OSS 20B/120B, DeepSeek v4 Flash, and briefly Claude Sonnet 4.6. The public repository is workofart/harness-experiment.
The system nests three loops. An outer self-improvement loop, an experiment loop over benchmark tasks, and individual task runs. Two distinct agents operate: an Improvement Agent that edits the harness, and an inner Task LLM that solves benchmark tasks using whatever harness the Improvement Agent has produced. The framing offered is an "IDE to human developer" relationship: harness customizations help weaker models but burden stronger ones.
Prompt-only self-editing failed. The opening attempt hard-coded task-specific hints and raised the reasoning budget despite explicit instructions not to. Code grew 55% with no plateau while solve counts did not improve; the author characterized the growth as corrective policy rather than new task-solving capability. Accumulated rules produced a "policy maze" the Task LLM navigates instead of solving the actual task.
Every rule is described as a "tax." A schema-validation rule that helped weak models broke Sonnet 4.6, which failed a task after seven rejections. The guiding heuristic offered is that a customization is worth keeping only "if the customization removes more uncertainty than it adds." Cost crept from $2-3/day to $10-15/day over the run.
| Pattern | Purpose |
|---|---|
| Statistical promotion gates | Binomial test against a pooled baseline rather than naive aggregate solve scores |
Persistent working memory (learning.md) |
Record bottlenecks, exhausted mechanisms, and partial wins to shrink the search space |
| Deterministic supervisor | Remove experiment-control decisions from the agent; physically restrict which files are editable |
| Sparse git checkout + worktrees | Bound context pollution and enforce edit boundaries |
| Prefix-cache-friendly state | Append-only message turns instead of rebuilt blobs; lifted cache hits from ~13% to ~72% |
| Provider routing | Pin preferred providers for cache consistency |
Placed in Techniques / Assess / center.
The central claim is a cautionary one: making self-improvement compound without human supervision is hard, and most autonomous harness edits add policy complexity without raising capability. The author explicitly frames results as intuitions and heuristics, notes the binomial promotion gate is "a promotion heuristic, not a formal scientific claim," and ends on open questions and future steps. No production use. Center rather than inner because the work reads as a reproducible warning with concrete metrics rather than a recommended practice; the transferable value is the supporting patterns (deterministic supervisor, statistical gates, append-only cache-friendly state), several of which stand on their own independent of the self-improvement premise. Distinct from radar/techniques/MaintainabilitySensorsForCodingAgents, which instruments a codebase so a human-supervised agent gets quality feedback; this technique removes the human from the improvement loop entirely and reports that the loop tends to degrade without a deterministic supervisor constraining it.
Trial gate: an autonomous harness-improvement loop run against a held-out task panel with a statistical promotion gate, showing solve-rate gains that persist on unseen tasks in a real project.
