v0.27.0 — a check after an action is judged

This release closes a way a step could pass on the agent's word alone. Steps that act and then check are judged differently, which is why this is a minor.

A step that acts and then checks has its check judged

A step like "save the transaction by clicking "Add" and verify the first transaction has payee "Home Depot"" could end with the agent answering done after the click. The step then closed on the agent's own sentence about the page, and nobody judged it. On a real budgeting app's suite, 4 of 61 such steps closed this way, and on the demo app 5 of 35 on gpt-6-luna (#172).

A check verb after and, then, a comma or a semicolon now makes that done a claim the judge decides, as for an assertion. After the change, every such step was judged. The budgeting app's suite still passed 21 of 21, and the run cost about the same number of calls.

Upgrade note: a step of this shape that passed before only because nobody checked it can now fail. That failure is the point.

Reliability, measured again

The README's table was measured again on this change: no false FAIL and no false PASS on either reference model. The one false FAIL Haiku had on 0.26.0 did not recur.

Documentation

• An editable data grid needs one cell per step. The README shows a grid test written one cell per step in keyboard order. Written as one step, the same test failed 3 runs in 3; written this way, it passed 3 in 3 (#67).
• Jev measured as a judge. TypeSafe AI's decision model was replayed on the judge corpus (#174). Corrected after release: these notes first reported 34 of 41 cases right and 6 false PASSes. That run gave Jev the step without the [redacted NAME] rewrite and without the label check every LLM verdict gets. Measured with both, it got 40 of 41 and no false PASS. It writes no reason for a failed step, which the agent's retry reads, so it is not a drop-in judge. The write-up has the numbers and a script to measure a later version for about a cent.

Full entry in the CHANGELOG.

v0.26.0 — pages that are still rendering, dialogs past the cap, and repeated clicks

This release fixes four ways a test failed on a correct page, or passed a run that never happened. Each was found running blastproof against a real budgeting app. Steps execute differently, which is why this is a minor.

A step waits for a page that is still rendering

An app that renders from data already in the browser goes network-idle before its first paint is done. A step could read the half-built page, and the model failed it with nothing done: no "Try the demo" button, only a "Set up sync" link, under concurrency: 4 (#14). A fail with nothing done now waits up to 3 s for the page to change, once per step. A changed page goes back to the model. A still page fails as before, with no extra model call. On a page that renders its button 2 s after idle, the step failed 3 runs in 3 and now passes 3 in 3.

The open dialog survives the snapshot cap

Frameworks render dialogs, menus and command palettes at the end of the page. A snapshot over max_snapshot_lines kept only the top, so the element the last action opened was the first thing dropped. On a 670-line page, the command palette was at line 606, and both of its tests failed at the default cap. A cut now keeps the open dialogs and the focused element first, and both pass. Omitted lines are marked where they were, and a step that fails on a cut snapshot names max_snapshot_lines in its reason (#162).

"Click Next until…" works

Within a step, a click that had already succeeded was always refused, so "click Next until the tour shows its last step" failed on its second click. A repeat now goes through when it directly follows itself and the page changed after the last press. Every other repeat is still refused, including a form refilled and submitted again. An onboarding tour now passes 3 in 3, with all 7 clicks (#164).

No score when nothing ran

When the provider refused the first call, or the budget ran out during sign-in, the reports and the action's score output said 100, even though the run exited 1. That run now has no score: the JUnit report omits it, the HTML report shows none, and the console says Score: none (#134).

Upgrade note: a workflow step reading outputs.score gets an empty string for that run.

Also

• A call that the run's deadline aborts is reported as the deadline, even when its timer fires a millisecond early (#159).
• The reliability table in the README was measured again on this release. On Haiku, one false FAIL in 40, in the notes test; everything else is unchanged.

Full entry in the CHANGELOG.

v0.25.0 — measured defaults, and the element the model read

This release changes which model a config with no model uses, makes an action land on the element the model actually read, and bounds every model call. Defaults and execution change, which is why this is a minor.

Default models, chosen by measurement

With no llm.model set, openai now uses gpt-6-luna (it was gpt-4o-mini) and ollama uses gpt-oss:20b (it was qwen2.5). On the reliability benchmark, the old defaults failed almost every test on the correct demo app: 35 of 40 for gpt-4o-mini, 40 of 40 for qwen2.5. gpt-6-luna failed none and gpt-oss-20b failed 2. The table for nine models is in Choosing a model (#154).

Upgrade note: on Ollama, run ollama pull gpt-oss:20b (about 14 GB) before running a config with no model. To keep the old default, set model: qwen2.5.

An action lands on the element the model read

The runner used to search the page for the model's description of an element and take the first match. A target with a role and no name clicked the first button on the page (#132). Two controls sharing a name resolved to the first one (#60). When such a click succeeded, the log could not show it.

Every snapshot now gives each element a ref, and an action names that ref. A ref resolves to that element or to nothing. The role and name the model gives are checked against it, and the log names the element actually clicked. On the two reference models, 5 samples each, verdicts did not change and runs used about 10% more tokens.

Every model call is bounded

• Output: every call asks for at most 4096 output tokens. Without a limit, open-weight models ran whitespace up to 131k tokens, and an account with credit left was refused for the reservation (#126).
• Time: each call has a timeout, llm.timeout_s (300 s, or 900 s for Ollama), and the run's deadline aborts a call in flight (#133).
• Spend: the tokens of an unusable answer are counted (#136).

Also

• llm.extra_body sends a gateway its own options. The docs include the OpenRouter recipe for routing open-weight models around providers that ignore the JSON schema (#150).
• New Action name. The Action is now named blastproof e2e AI testing, so it can be listed on the GitHub Marketplace. uses: hamc/blastproof@v0.25.0 and its inputs are unchanged.
• docs/ci.md installs the right CLI. The example now installs the CLI it documents; since 0.12.0 it had installed 0.11.0.

Full entry in the CHANGELOG.

v0.24.0 — measuring how often a verdict is wrong

This release starts measuring how often a verdict is wrong, and fixes what the measurement found. Verdicts change, which is why this is a minor.

How often a verdict is wrong

npm run eval:reliability runs the dogfood suite on the demo app as it is, and each test on mutants: copies of the app with one bug seeded. A test that passes on a mutant is a false PASS. A test that fails on the correct app is a false FAIL. Some mutants change the value a step quotes. Others reproduce the shape of a wrong verdict this project already had.

After this release's fixes, 5 samples per model:

	claude-haiku-4.5	gpt-6-luna
false FAIL, correct app (40 runs)	0 (≤ 8.8%)	0 (≤ 8.8%)
false PASS, literal bugs (30)	0 (≤ 11.4%)	0 (≤ 11.4%)
false PASS, subtle bugs (20)	0 (≤ 16.1%)	0 (≤ 16.1%)

The bounds are 95% upper bounds. The demo app is small and well labelled, so these numbers flatter, and a real application will do worse. The point is that anyone can rerun them, and that every change claiming better verdicts now has to move them.

A value in the wrong place no longer passes

The first run of that table found a false PASS on Haiku, the Anthropic default. The confirmation showed order #BP-1002, another paragraph mentioned #BP-1001 as a previous order, and "Verify the order number "#BP-1001" is displayed in the confirmation message" passed in 3 of 13 runs. A recording of the judge's input showed why: the executor's claim had called the previous-order paragraph "the confirmation message". A place the step names is now part of its outcome, and a claim cannot rename it. That gave 0 wrong passes in 10 runs, and the table above (#147).

A verification step only looks

On a page that did not show the message a step checked for, both reference models clicked "Add to cart" to make it appear, and the step passed. A step that starts with verify, check, confirm, ensure, assert, expect or validate may no longer click or press Enter, and it closes only on a passing judgment (#139).

Behaviour change: a "verify" step that relied on an action now fails on the refusal. Put the action in its own step, or start the step with it: "click Save and verify the note is listed".

A secret typed earlier counts

Since 0.23.0, a step like "log in with {{env.PASSWORD}} and verify the welcome heading" passes without typing when the login already happened. The check that fails a step naming a secret the page does not show then failed it. A variable used in an earlier step of the same test now counts (#141).

Known issues

These are mostly about open-weight models, the ones provider: ollama runs, found while starting to measure them:

• every call reserves the provider's maximum output (#126), and some reasoning models emit whitespace up to it, 65k–131k tokens;
• a model call has no timeout of its own (#133), so those runaways end only at the HTTP client's 5-minute limit;
• through a gateway, open-weight models can reach providers that ignore the JSON schema, with no setting to avoid them (#150).

Full entry in the CHANGELOG.

v0.23.0 — the failure is no longer blamed on the wrong thing

Four fixes, all about a failure being blamed on the wrong thing. Verdicts change, which is why this is a minor.

A step that was already done no longer fails

"Dismiss the cookie dialog by clicking "Me want it!" and verify it is gone" failed on pages where an earlier step had already dismissed the dialog. The judge read the verb as a requirement: "the dialog's absence cannot be verified as the result of dismissal rather than never being shown." Neither the executor's claim nor dropping "by clicking" from the step moved it. Against OWASP Juice Shop, the first judgment of that step failed in 4 of 5 runs, and the test passed only because the model kept asserting until one got through.

The judgment now states the step's outcome first, as a sentence with the action taken out, and decides that sentence. The same step now passes on its first judgment in 5 of 5 runs. Three new cases in the judge corpus check the dangerous direction: the dialog still there, a different dialog closed, the wrong dialog still open. They fail every time, as they should.

What this costs, on purpose: the means is not checked. "Sign in with SSO" passes for a user signed in some other way. A snapshot records a state, not how the page got there. Where the means matters, the step names what the means leaves on the page.

When the provider refuses, the application is not blamed

When the OpenRouter balance ran out mid-run, every following test failed as if the application had regressed. With a login configured, the run said Authentication failed, exit 2, and sent the reader to the auth recipe. A billing refusal, a revoked key and an outage were all treated as a malformed answer from the model: retried, then failed.

A call that gets an HTTP status of 400 or above, or no answer at all, now ends the run the way an exhausted budget does. That holds after the provider SDK's own retries. The tests that finished keep their results, the rest are not run, the run exits 1 whatever --min-score says, and the message says what happened and what to do:

Run incomplete: model provider refused the request (HTTP 402): This request requires more credits… Add credit to the provider account, then run again.

A garbled judgment is an attempt, not a verdict

One unparseable answer from the judge failed the step on the spot, and during a login it ended the run with exit 2. The spec has always required that a malformed answer costs one attempt, and the action call already met it. The judge call never did, because it sat outside the code that counts attempts. It now counts, and the login's verify is asked again.

A duplicate click no longer slips through

contained-recovery refuses to repeat a click or submit within a step, so a retry cannot write a second row. The identity it compared included the button's visible text. A model that added the text on its second attempt got a "new" action and clicked again. A commit is now identified by what resolves it: role, name (ignoring case and spacing) and value. Duplicate writes in the demo suite went from 2 runs in 5 to none.

Known issues

An adversarial QA pass against Juice Shop before this release found no regression in these changes. It did find these, now filed:

• a target with a role but no name clicks the first element of that role (#132);
• a model call has no timeout, so a provider that stops answering holds the run past --max-duration (#133);
• an incomplete run in which nothing executed reports a score of 100, including in the Action's score output (#134);
• a call that fails inside the AI SDK's own parsing is not counted against the budget (#136).

Full entry in the CHANGELOG.

v0.22.0 — 0.21.0 fixed the form nobody writes

One fix, and the corpus that verified it. It changes verdicts, which is why this is a minor.

0.21.0 fixed the form nobody writes

0.21.0 labelled each redaction with its variable, so a step verifying one secret stopped passing against a page showing another. It was verified with the secret's value written literally in the step, which the mask turns into a label on both sides. People don't write tests that way. They write the placeholder:

- open the Account menu and verify it shows the account email {{env.TEST_OTHER}}

A placeholder isn't a value, so the mask leaves it alone. The judge read {{env.TEST_OTHER}} in the step and [redacted TEST_EMAIL] on the page, and had to work out by itself that those were different secrets. Against a live OWASP Juice Shop, a test built to fail this way passed in 3 of 5 runs. The executor offered a bridge, "[redacted TEST_EMAIL] which is the account email {{env.TEST_OTHER}}", and the judge took it every time its exact input was replayed.

A second path ran beside it. The judgment schema still asked whether the page "satisfies the expectation", the executor's claim, not the step. So when the executor offered "redirected away from login, or an Account menu is accessible", the judge passed on the easy half.

Three changes, because the first two weren't enough

• One vocabulary. The judge now reads every placeholder in the step, the claim and the record as the label the page carries. The executor still sees and types placeholders.
• The right question. The schema asks whether the snapshot shows the step's outcome, and produces its reason before its verdict instead of after.

Measured on the captured inputs, neither half worked alone; together they closed both paths. Live, they halved the rate and stopped there. Given [redacted TEST_OTHER] in the step and [redacted TEST_EMAIL] on the page, in exactly the vocabulary that was meant to settle it, the model still wrote "which matches" in 2 of 4 runs.

• A check on our own tokens. After the model's verdict, a PASS on a step naming [redacted X] is failed when X is neither on the page nor in the step's own actions while another label is on the page. Labels are what the mask writes, so this compares the tool's own output rather than reading prose. It only ever turns PASS into FAIL. The record clause keeps a step that typed a secret into a form it then submitted, and the other-label clause keeps a step asserting that a secret is absent.

On the final code, six live runs: the test built to fail never passed. In one of them the model passed the step twice and the check failed both.

One cost, stated rather than discovered: a step asserting a secret is absent, on a page that shows a different secret, now fails, with a reason naming both labels.

Why there is now a corpus

Asked whether these fixes were going in circles, I counted. Nine changes have touched the verdict since July, each closing a distinct failure found from outside, and none was traced to an earlier fix. The most plausible candidate was tested and ruled out. What was true is that each was verified only against its own reproduction, plus a dogfood suite with none of these cases in it. That's exactly how 0.21.0 missed the placeholder form.

evals/judge/ now holds 33 judge inputs with known verdicts from every incident that produced a wrong one (#31, #35, #87, #112, #120, #121). They're captured from real runs where possible, marked as reconstructed where not, and contain only masked text. npm run eval:judge replays them through the real judge. It needs a key, so it isn't in CI; changing the judge now means running it first. #121's cases are in it, marked as known failures, and the run will say when they stop failing.

Found on the way

Verifying this surfaced three things now filed: a repeated commit that slips past contained-recovery when the model adds a button's text, writing a duplicate (#124); a provider refusal, such as an exhausted balance, failing tests as if the application were broken (#125); and every call reserving 64000 output tokens (#126).

v0.21.0 — every secret looked like every other

Two fixes, both about secrets written into a test, and one of them a wrong PASS. Both change what a run or a draft does rather than only what it prints, which is why this is a minor.

Every secret looked like every other

Since the mask existed, every registered {{env.*}} value was redacted to the same ***. That kept values out of prompts and reports, which is its job. It also erased the one thing a judgment on a secret needs: which secret it was.

The judge sees the step through the mask, the same as the page. So a step verifying text equal to a secret reached it as "***", the page showed "***", and the judge was right to call them equal, whatever the page actually said. #87 was filed as the mild half of this, "such a step can never pass". Measured, it passes, and it also passes when it's false:

step verifies	page shows	before	now
Unknown promo code "HUNTER2".	Unknown promo code "HUNTER2".	3/3 PASS	3/3 PASS
Unknown promo code "SAVE99".	Unknown promo code "HUNTER2".	3/3 PASS	3/3 FAIL

Both HUNTER2 and SAVE99 were registered secrets. For a tool whose output gates merges, a deterministic PASS on a wrong page is the worst thing it can say.

A redaction now names its variable:

- status: Unknown promo code "[redacted PROBE_SECRET]".

The label carries the name and nothing derived from the value: no length, no prefix, no hash, because each of those narrows the value and a hash of a short password is a lookup. The same secret always gets the same label, whichever form of it matched (an uppercased echo, a percent-encoded URL), and two secrets get two. After the change the judge explains its FAIL itself: "these are two different redacted values".

The same change settles the opposite complaint. A value put in {{env.*}} for portability rather than secrecy (a tenant, a test account) used to be unassertable. It is now asserted on by identity, without being unmasked.

One door had to close with it. The executor only admits a typed value that came from the step or from the page the model read, and the page it read is the masked one. A label is on it, so without a guard the model could copy [redacted TEST_EMAIL] into a field and have it typed literally. A fill or select containing a label is now refused, with an explanation the model can act on.

Output change: logs, JUnit and the HTML report show [redacted NAME] where they showed ***. Anything that matched *** in those files needs to match [redacted instead.

"Never write a password, token or key"

That was the planner's rule, and an email address is none of the three. So a login draft used {{env.TEST_PASSWORD}} for the password and a literal address for the account, and it looked right to the person reviewing it, because the line that mattered most was handled.

The rule now names the property: any value that identifies an account or a person, with email, username and account number as examples, not as the list. On the demo's login page, which prints its own credentials as a hint:

prompt	drafts using a placeholder
old	0 of 5
new	4 of 5

With the old prompt every draft copied the hint, password included, which the old rule already forbade. That led to the second half of this fix. The planner's secret check only fired on a quoted value, and a model doesn't quote: all five of those drafts wrote Fill the Password textbox with demo123 and were accepted. A step naming a credential now counts as carrying a value when it's quoted, or when it enters a named value by the definition run's authoring check already uses. Replayed offline, all six literal-password drafts from the measurement are refused, and the four with placeholders aren't.

plan also warns when a draft writes an email address the page it was drafted from doesn't show. The draft is still written: a literal address is a wrong test, not a leak, and the reviewer is who should fix it. An address the page does show isn't reported, since verifying a displayed contact is legitimate. That means a page printing its own credentials hides a copied one, and the authoring reference says so rather than implying otherwise.

Verified before tagging

• The #87 experiment was rerun unchanged after the fix: true 3/3 PASS, false 3/3 FAIL, and neither value appears in any log.
• Mutations on both fixes, each caught by at least one test. One of them first caught a hole in the tests themselves: removing the quoted-value branch turned nothing red, because the only quoted test was also an entry step. A test now pins a quoted secret in a step that enters nothing.
• The Dogfood workflow on the merge base, in CI: 8 of 8, Score 100 against --min-score 80, 93 model calls. Worth saying plainly: that suite uses no {{env.*}} value, so it shows the prompt changes didn't regress the common path. It doesn't validate the label. The experiment above does.

Also

#112, the wrong PASS filed during 0.20.0, was measured 20 times after the demo's CSS fix and didn't reproduce. It's closed as not reproduced, with the data on the issue.

v0.20.0 — the report said *** and showed the value

Two secret leaks. Both were in the part of the tool that exists to keep a secret out of what leaves your machine, and both were silent — which is why this is a minor rather than a patch.

The report said *** and showed the value

{{env.*}} values were redacted from every text channel. The HTML report then embedded the failure screenshot as a base64 data: URI, and nothing masks pixels:

report text contains the secret in the clear?  no (masked)
report embeds a PNG as a data URI?             YES
size of the embedded PNG:                      26 KB

That 26 KB showed the value twice: in the field it was typed into, and in the page's own status line. A type="password" field renders as dots and is safe by accident; an email in a profile header, a token in a "copy this" box, a one-time code echoed into a confirmation are not.

What makes it worse than a file on disk is what the file is for. The report is self-contained precisely so it can be attached to a pull request, uploaded as a CI artifact and forwarded to someone who was not there. A screenshot is captured only on failure — exactly when a report gets shared. .blastproof/reports/ is git-ignored, which protects the repository and does nothing for the copy in the artifact bucket.

When anything in a run references {{env.*}}, the report now links to each failure screenshot instead of carrying it. The PNG stays where it always was, and on that path the file is never read, so no later edit can put the bytes back by accident. A run that referenced no {{env.*}} value is unchanged and still opens offline with its images.

The condition is the run's mask, not the failed test's own placeholders: a login credential is typed once and then echoed by the pages of every test that follows. That costs most authenticated suites the inline image, and the link is one click from it.

Our own CI guide was part of the leak. It recommended uploading report.html and .blastproof/reports/, which is where the unmasked PNGs live. The example now uploads the report alone and says what that directory contains.

The mask compared byte-for-byte

The second leak needs no screenshot. SecretsMask replaced by escaped literal, case-sensitively, so the guarantee held only while the string on the page was byte-identical to the one in the environment. Applications reformat what they echo — ours does:

supplied	the page showed	before	now
HUNTER2	Unknown promo code "HUNTER2".	"***"	"***"
hunter2	Unknown promo code "HUNTER2".	leaked	"***" + warning
Hunter2	Unknown promo code "HUNTER2".	leaked	"***" + warning

The leaking run and the masked one printed the same score, the same exit code and the same output. Nobody could tell them apart without reading the report by eye.

The comparison is now the value case-insensitively, with runs of whitespace matched as runs of whitespace. That is not a new rule: it is the normalization blastproof already applies in the other direction, when deciding whether a value the model typed came from the page.

It is deliberately the whole rule. There is no catalogue of application-side transforms, and a value your app base64-encodes, hashes or truncates is still not recognised — pinned by a test, so the boundary is checked rather than assumed. An enumeration never closes, and one that implies completeness stops people being careful, which is worse than a limit stated plainly.

What ends is the silence where detection is possible. When a redaction is found only by the widened comparison, the run says so once per variable:

warning: the application returned PROBE_SECRET in a different form than the value supplied
(case or spacing). It was redacted, but a form this tool cannot recognise — an encoding,
a hash — would not have been. Check what that page does with the value.

It names the variable and neither form of the value, and it changes no verdict: a page's formatting is not a test failure.

One cost worth stating. Wider masking redacts a little more of a page, and redacted text cannot be asserted on. The advice in the README has not changed — do not write a step that asserts on an {{env.*}} value — it just matters slightly more.

Verified before tagging

Both leaks were reproduced against examples/demo-app with a real browser, and both fixes were then confirmed by an agentic run with a real model: the report carries no image and its link opens the PNG; the lowercase run warns, the uppercase one does not, neither leaks the value into stdout, stderr or the report, and both reach the same verdict.

Each fix was mutated to check the tests can see it: three mutations on the report path turn 3, 1 and 4 tests red; four on the mask turn 2, 4, 7 and 2 red.

The full dogfood ran on the merge base — 8 of 8, Score 100, 95 model calls against a real browser and a real model. Worth recording honestly: that suite uses no {{env.*}} value anywhere, so it proves nothing regressed on the common path and is not what validates either fix.

Also

examples/demo-app always rendered its discount row, because a display: flex rule beat the hidden attribute. Not shipped code, but it is what this project tests itself against, and a fixture that puts false evidence of success on the page produced exactly that — a wrong PASS, now filed as #112.

v0.19.0 — a login nothing selected needed

Two fixes. Both change what a run does rather than only what it prints, which is why this is a minor.

A login nothing selected needed

The session had one producer and many consumers, and only the consumers ever asked whether it was wanted.

// the producer — "is a login configured for this project?"
if (config.auth) {

// a consumer, thirty lines earlier — "does this test want one?"
const context = await browser.newContext(test.auth ? contextOptions(session) : {});

Those are different questions, and the producer was asking the wrong one. So a run whose every selected test declared auth: false performed the entire login journey anyway — and said so itself:

$ blastproof run --tag consent --dry-run
Dry run: 1 test(s) selected
Worst case: up to 126 model call(s) for this selection, including the login journey

Including the login journey, for the one test that had declared in writing that it did not want one. With nothing selected at all the ceiling was 84, all of it login.

The run now asks whether some selected test wants the session, from the selection it is about to execute. One authenticated test is enough to bring the login back; there is no partial login to perform.

It was never only cost

This was filed as waste. That framing was wrong, and two paths are why.

A tight budget turned it into a verdict. The login spends from the same allowance the tests do, and exhausting it marks the run incomplete — which exits 1 regardless of --min-score, by design, because exhaustion stops a run rather than failing a test. A suite whose selected tests had all opted out could therefore exit 1 over a session none of them would read.

A missing session file was a deterministic exit 2. auth.storage_state is read from disk before any test runs, and an unreadable file aborts the run. That happened whether or not anything in the selection would have opened it.

This is why the rule covers every strategy, not only auth.steps. Skipping only the one that spends model calls would have fixed the bill and left the two failures in place.

The ceiling moved with it

--dry-run counted the login unconditionally, on the principle that a ceiling must not be exceedable by anything it claims to bound. Left alone, it would have become wrong in the other direction. It now reads the same selection the run does:

	before	after
run --tag consent --dry-run	126	42
a selection of nothing	84	0

A ceiling that keeps charging for work that will not happen is conservative, and nothing exceeds it. But an overestimate that persists teaches people to discount the number, which is how a ceiling stops being read at all.

What it actually saves

Measured live against this repository's demo app — same selection, same model, the predicate the only difference:

run --tag consent	model calls	tokens
before	18	30,167
after	7	12,136

Both runs passed with Score 100. The saving is the login not happening, not less testing.

The value concentrates where the selection is small, which is to say under --impacted: the tighter the diff, the larger the share of a run that used to belong to a session nobody asked for.

What does not change

A suite that never mentions auth: behaves exactly as it did — the field defaults to true, so most users will never see this. plan still authenticates whenever a recipe is configured, and is pinned by a test that fails if this rule ever leaks into it: it is generating the tests, so it has no selection to consult, and a planner that snapshotted the login wall would draft tests describing the login wall.

No new flag. The information was already in the selection, and asking for it would be asking people to compute what we can read.

--impacted says what the diff did not see

The second fix. --impacted selects from git diff <base>...HEAD, which compares two commits — so uncommitted work was invisible to it. A run against a dirty working tree reported no affected routes, selected nothing, and exited 0 without ever saying what it had not looked at. --fail-on-unmapped could not catch it either, because that guard inspects the changed files the hole had already excluded it from.

The working tree is now read wherever the diff is computed, and every change the diff excluded is named on stderr with the routes it maps to. Two classes stay silent so the warning stays worth reading: a file ignore: already declares irrelevant, and a file already in the diff.

Verified before tagging

The dogfood ran the full suite against a real browser and a real model on the change's branch, before it merged: 8 of 8, Score 100 against a --min-score 80 gate, 95 model calls. The login was performed exactly once for the selections that wanted it, and not at all for those that did not.

Worth recording honestly: the two failure paths this closes — the budget exhausted mid-login, and the unreadable storage_state — are proven by unit tests and by mutation, not by a live run. Neither has a cheap live reproduction, and both were reasoned from the code before they were tested. The mutations are what give that weight: changing some to every turns two tests red, dropping the ceiling's half of the predicate turns two red, and reverting the run's half turns four red, with nothing outside the new block moving.

v0.18.0 — a step the agent could not carry out was passing

One fix, and it is the kind worth interrupting a release cadence for: the tool could report a perfect score on a suite where a step accomplished nothing.

A step the agent could not carry out was passing

The runner closed a step whenever the agent said it was finished. It never asked whether anything had been done.

So a step naming a control the page does not have came back green — while the agent's own explanation, printed right there, said the opposite:

step 2/2: enter a value in the promo code field
    -> done :: The current page does not contain a promo code field,
               so this step cannot be completed.

PASS    P0    Score: 100

That step then contributed its full priority weight to the score --min-score gates merges with. This is not a suite anyone notices failing — it is an inflated number, silently, on the one the gate trusts. A tool whose job is to stop a broken change reaching production was telling people their broken change was fine.

A step now closes on what was done in it.

The same rule catches a step that verified nothing. A verification could previously finish having made no assertion at all and still weigh on the score. That was a separate open defect; it turned out to be the same one seen from the other side.

The part with a cost, said plainly

There is such a thing as a legitimately empty step — dismissing a banner that is not there. And from the agent's side, "there was nothing to do" and "I could not do this" are the same sentence. The runner cannot separate them and does not try.

So what it asks for changed. The agent may no longer declare a step complete; it has to show the outcome — assert that the banner is absent, and let that assertion be judged against the page. Self-report became evidence.

A defensive step of that shape now costs one extra model call. That is the price of the score meaning something, and it belongs in plain sight rather than in a diff.

Verified before tagging

The dogfood ran the full suite against a real browser on this commit: 8 of 8, Score 100, 92 model calls. The change sits in the loop's termination condition, so every step of every test passes through it.

Worth recording honestly: across four live probes the new refusal never actually fired — faced with an impossible step, the model asserts rather than declaring, and fails on the assertion budget instead. The unit tests and a mutation are what prove the rule; the live runs prove it does not disturb the ordinary path.

One thing that surprised us

Fourteen of the project's own tests had to change. None of them was about this behaviour — they were about setup ordering, origin boundaries, timing budgets — and every one used a bare "finished" as a convenient way to end a step.

That is a better measure of how common the shape is than any survey we ran: it is the intuitive thing to write, and it was the thing the runner had been accepting without question.