v0.23.0 — the failure is no longer blamed on the wrong thing

Four fixes, all about a failure being blamed on the wrong thing. Verdicts change, which is why this is a minor.

A step that was already done no longer fails

"Dismiss the cookie dialog by clicking "Me want it!" and verify it is gone" failed on pages where an earlier step had already dismissed the dialog. The judge read the verb as a requirement: "the dialog's absence cannot be verified as the result of dismissal rather than never being shown." Neither the executor's claim nor dropping "by clicking" from the step moved it. Against OWASP Juice Shop, the first judgment of that step failed in 4 of 5 runs, and the test passed only because the model kept asserting until one got through.

The judgment now states the step's outcome first, as a sentence with the action taken out, and decides that sentence. The same step now passes on its first judgment in 5 of 5 runs. Three new cases in the judge corpus check the dangerous direction: the dialog still there, a different dialog closed, the wrong dialog still open. They fail every time, as they should.

What this costs, on purpose: the means is not checked. "Sign in with SSO" passes for a user signed in some other way. A snapshot records a state, not how the page got there. Where the means matters, the step names what the means leaves on the page.

When the provider refuses, the application is not blamed

When the OpenRouter balance ran out mid-run, every following test failed as if the application had regressed. With a login configured, the run said Authentication failed, exit 2, and sent the reader to the auth recipe. A billing refusal, a revoked key and an outage were all treated as a malformed answer from the model: retried, then failed.

A call that gets an HTTP status of 400 or above, or no answer at all, now ends the run the way an exhausted budget does. That holds after the provider SDK's own retries. The tests that finished keep their results, the rest are not run, the run exits 1 whatever --min-score says, and the message says what happened and what to do:

Run incomplete: model provider refused the request (HTTP 402): This request requires more credits… Add credit to the provider account, then run again.

A garbled judgment is an attempt, not a verdict

One unparseable answer from the judge failed the step on the spot, and during a login it ended the run with exit 2. The spec has always required that a malformed answer costs one attempt, and the action call already met it. The judge call never did, because it sat outside the code that counts attempts. It now counts, and the login's verify is asked again.

A duplicate click no longer slips through

contained-recovery refuses to repeat a click or submit within a step, so a retry cannot write a second row. The identity it compared included the button's visible text. A model that added the text on its second attempt got a "new" action and clicked again. A commit is now identified by what resolves it: role, name (ignoring case and spacing) and value. Duplicate writes in the demo suite went from 2 runs in 5 to none.

Known issues

An adversarial QA pass against Juice Shop before this release found no regression in these changes. It did find these, now filed:

• a target with a role but no name clicks the first element of that role (#132);
• a model call has no timeout, so a provider that stops answering holds the run past --max-duration (#133);
• an incomplete run in which nothing executed reports a score of 100, including in the Action's score output (#134);
• a call that fails inside the AI SDK's own parsing is not counted against the budget (#136).

Full entry in the CHANGELOG.