Skip to content

Harness Engineering

Jump to bottom
pierry edited this page Jun 15, 2026 · 2 revisions

Harness Engineering

The theory harness-kit is built on. Read this to understand why the repo is shaped the way it is, guides, sensors, evals, gated stages.

Agent = Model + Harness

The term harness is shorthand for everything in a coding agent except the model itself.

Agent = Model + Harness

The model generates. The harness is the scaffolding around it: the prompts and guides that steer it before it acts, the tools it can call, the checks that catch its mistakes after it acts, the permissions, the memory, the context it sees. You usually cannot change the model. You can change the harness. Harness engineering is the practice of improving the harness to raise the probability of good output and to let the agent self-correct before a human ever sees the result.

Source: Birgitta Böckeler, Harness engineering for coding agent users, in Martin Fowler's Exploring Gen AI series, martinfowler.com (2026). The companion piece, Maintainability sensors for coding agents, develops the sensor half.

Feedforward and feedback

The harness has two kinds of control, borrowed from control theory.

  • Feedforward controls (guides) anticipate the agent's behavior and steer it before it acts. Examples: a writing-style guide, a template, a coding convention, an example of good output. In harness-kit these are the guides/.
  • Feedback controls (sensors and evals) observe after the agent acts and help it self-correct. Examples: a linter, a test, a structure check, a scored review. In harness-kit these are the sensors/ and evals/.

Feedback is most powerful when its signal is optimized for LLM consumption: a sensor that does not just say "failed" but says "missing section X; add it with these fields" gives the agent exactly what it needs to fix itself in the next turn.

Computational and inferential controls

Cutting the other way, controls are either:

  • Computational (deterministic): linters, tests, schema checks, structure rules. Same input, same verdict, every time. Cheap, fast, exact, but blind to meaning. harness-kit's sensors are computational.
  • Inferential (LLM-based): semantic judgment: "is this PRD clear?", "does this design name its trade-offs?". Handles meaning that no regex can, but is probabilistic and must be calibrated. harness-kit's evals are inferential.

You need both. A structure check cannot tell you the prose is vague; an LLM judge cannot be trusted to deterministically enforce "exactly one H1 heading". The discipline is to push everything you can make deterministic into a sensor, and reserve the eval for genuine semantic judgment.

What the controls regulate

Böckeler frames three dimensions a harness regulates:

  1. Functional behavior: does it do the right thing? (tests, acceptance criteria)
  2. Maintainability: is it clean, simple, conventional? (linters, structure, style)
  3. Architecture fitness: does it fit the intended design and constraints? (fitness functions, convention overrides)

harness-kit's gates touch all three: sensors enforce structure and conventions (maintainability, fitness), evals score clarity and rigor (functional intent), and the per-repo conventions/ files pin architecture fitness.

Humans outside, in, or on the loop

Three postures for human involvement, by how the human relates to the agent's work:

  • Human outside the loop: fully autonomous; the agent ships without review. Rare, high-risk.
  • Human in the loop: the human reviews each individual output. Safe, but does not scale: your review rate caps the agent's throughput.
  • Human on the loop: the human maintains and improves the harness (the guides, sensors, evals) rather than inspecting every output. The harness reviews the outputs; the human reviews the harness.

"Humans on the loop" is the only posture that scales with agent throughput, and it is the posture harness-kit is designed for. You do not hand-approve every artifact's prose; you tune the rubric once and the rubric judges every artifact. When the agent drifts, you fix the guide, not the output.

How harness-kit instantiates it

Every stage of every pipeline is a small harness:

GUIDE     feedforward      how to write it          (guides/, templates/, examples/)
REF       context          what to pull in          (AGENTS.md, prior artifacts, conventions/)
SENSOR    deterministic    must-pass structure      (blocks approval, regex via sensor-runner.py)
EVAL      inferential      scored rubric            (LLM-judge, threshold 8.0, retry x3)

The artifact only advances when the sensor passes and the eval clears 8.0. Nothing advances on vibes. This is the same split, computational + inferential, feedforward + feedback, applied to PRDs, PRPs, plans, code, tests, PRs, and (in the system-architect agent) System Design Docs.

The payoff of writing the harness in plain markdown (guides, sensors, evals are all markdown; only the runner and hooks are code) is that the harness is legible and bendable. You can read exactly what will be checked, and change it, without touching the model.

See also

References

  • Birgitta Böckeler, Harness engineering for coding agent users, martinfowler.com, 2026.
  • Birgitta Böckeler, Maintainability sensors for coding agents, martinfowler.com, 2026.
  • Martin Fowler, Exploring Gen AI (essay series).

harness-kit

The harness

v5: autonomy

System design

Back to repo →

Clone this wiki locally