v0.18.0 — a step the agent could not carry out was passing

One fix, and it is the kind worth interrupting a release cadence for: the tool could report a perfect score on a suite where a step accomplished nothing.

A step the agent could not carry out was passing

The runner closed a step whenever the agent said it was finished. It never asked whether anything had been done.

So a step naming a control the page does not have came back green — while the agent's own explanation, printed right there, said the opposite:

step 2/2: enter a value in the promo code field
    -> done :: The current page does not contain a promo code field,
               so this step cannot be completed.

PASS    P0    Score: 100

That step then contributed its full priority weight to the score --min-score gates merges with. This is not a suite anyone notices failing — it is an inflated number, silently, on the one the gate trusts. A tool whose job is to stop a broken change reaching production was telling people their broken change was fine.

A step now closes on what was done in it.

The same rule catches a step that verified nothing. A verification could previously finish having made no assertion at all and still weigh on the score. That was a separate open defect; it turned out to be the same one seen from the other side.

The part with a cost, said plainly

There is such a thing as a legitimately empty step — dismissing a banner that is not there. And from the agent's side, "there was nothing to do" and "I could not do this" are the same sentence. The runner cannot separate them and does not try.

So what it asks for changed. The agent may no longer declare a step complete; it has to show the outcome — assert that the banner is absent, and let that assertion be judged against the page. Self-report became evidence.

A defensive step of that shape now costs one extra model call. That is the price of the score meaning something, and it belongs in plain sight rather than in a diff.

Verified before tagging

The dogfood ran the full suite against a real browser on this commit: 8 of 8, Score 100, 92 model calls. The change sits in the loop's termination condition, so every step of every test passes through it.

Worth recording honestly: across four live probes the new refusal never actually fired — faced with an impossible step, the model asserts rather than declaring, and fails on the assertion budget instead. The unit tests and a mutation are what prove the rule; the live runs prove it does not disturb the ordinary path.

One thing that surprised us

Fourteen of the project's own tests had to change. None of them was about this behaviour — they were about setup ordering, origin boundaries, timing budgets — and every one used a bare "finished" as a convenient way to end a step.

That is a better measure of how common the shape is than any survey we ran: it is the intuitive thing to write, and it was the thing the runner had been accepting without question.