v0.24.0 — measuring how often a verdict is wrong

This release starts measuring how often a verdict is wrong, and fixes what the measurement found. Verdicts change, which is why this is a minor.

How often a verdict is wrong

npm run eval:reliability runs the dogfood suite on the demo app as it is, and each test on mutants: copies of the app with one bug seeded. A test that passes on a mutant is a false PASS. A test that fails on the correct app is a false FAIL. Some mutants change the value a step quotes. Others reproduce the shape of a wrong verdict this project already had.

After this release's fixes, 5 samples per model:

	claude-haiku-4.5	gpt-6-luna
false FAIL, correct app (40 runs)	0 (≤ 8.8%)	0 (≤ 8.8%)
false PASS, literal bugs (30)	0 (≤ 11.4%)	0 (≤ 11.4%)
false PASS, subtle bugs (20)	0 (≤ 16.1%)	0 (≤ 16.1%)

The bounds are 95% upper bounds. The demo app is small and well labelled, so these numbers flatter, and a real application will do worse. The point is that anyone can rerun them, and that every change claiming better verdicts now has to move them.

A value in the wrong place no longer passes

The first run of that table found a false PASS on Haiku, the Anthropic default. The confirmation showed order #BP-1002, another paragraph mentioned #BP-1001 as a previous order, and "Verify the order number "#BP-1001" is displayed in the confirmation message" passed in 3 of 13 runs. A recording of the judge's input showed why: the executor's claim had called the previous-order paragraph "the confirmation message". A place the step names is now part of its outcome, and a claim cannot rename it. That gave 0 wrong passes in 10 runs, and the table above (#147).

A verification step only looks

On a page that did not show the message a step checked for, both reference models clicked "Add to cart" to make it appear, and the step passed. A step that starts with verify, check, confirm, ensure, assert, expect or validate may no longer click or press Enter, and it closes only on a passing judgment (#139).

Behaviour change: a "verify" step that relied on an action now fails on the refusal. Put the action in its own step, or start the step with it: "click Save and verify the note is listed".

A secret typed earlier counts

Since 0.23.0, a step like "log in with {{env.PASSWORD}} and verify the welcome heading" passes without typing when the login already happened. The check that fails a step naming a secret the page does not show then failed it. A variable used in an earlier step of the same test now counts (#141).

Known issues

These are mostly about open-weight models, the ones provider: ollama runs, found while starting to measure them:

• every call reserves the provider's maximum output (#126), and some reasoning models emit whitespace up to it, 65k–131k tokens;
• a model call has no timeout of its own (#133), so those runaways end only at the HTTP client's 5-minute limit;
• through a gateway, open-weight models can reach providers that ignore the JSON schema, with no setting to avoid them (#150).

Full entry in the CHANGELOG.