Skip to content

v0.19.0 — a login nothing selected needed

Choose a tag to compare

@hamc hamc released this 13 Sep 10:50
· 53 commits to main since this release
a63229b

Two fixes. Both change what a run does rather than only what it prints, which is why this is a minor.

A login nothing selected needed

The session had one producer and many consumers, and only the consumers ever asked whether it was wanted.

// the producer — "is a login configured for this project?"
if (config.auth) {

// a consumer, thirty lines earlier — "does this test want one?"
const context = await browser.newContext(test.auth ? contextOptions(session) : {});

Those are different questions, and the producer was asking the wrong one. So a run whose every selected test declared auth: false performed the entire login journey anyway — and said so itself:

$ blastproof run --tag consent --dry-run
Dry run: 1 test(s) selected
Worst case: up to 126 model call(s) for this selection, including the login journey

Including the login journey, for the one test that had declared in writing that it did not want one. With nothing selected at all the ceiling was 84, all of it login.

The run now asks whether some selected test wants the session, from the selection it is about to execute. One authenticated test is enough to bring the login back; there is no partial login to perform.

It was never only cost

This was filed as waste. That framing was wrong, and two paths are why.

A tight budget turned it into a verdict. The login spends from the same allowance the tests do, and exhausting it marks the run incomplete — which exits 1 regardless of --min-score, by design, because exhaustion stops a run rather than failing a test. A suite whose selected tests had all opted out could therefore exit 1 over a session none of them would read.

A missing session file was a deterministic exit 2. auth.storage_state is read from disk before any test runs, and an unreadable file aborts the run. That happened whether or not anything in the selection would have opened it.

This is why the rule covers every strategy, not only auth.steps. Skipping only the one that spends model calls would have fixed the bill and left the two failures in place.

The ceiling moved with it

--dry-run counted the login unconditionally, on the principle that a ceiling must not be exceedable by anything it claims to bound. Left alone, it would have become wrong in the other direction. It now reads the same selection the run does:

before after
run --tag consent --dry-run 126 42
a selection of nothing 84 0

A ceiling that keeps charging for work that will not happen is conservative, and nothing exceeds it. But an overestimate that persists teaches people to discount the number, which is how a ceiling stops being read at all.

What it actually saves

Measured live against this repository's demo app — same selection, same model, the predicate the only difference:

run --tag consent model calls tokens
before 18 30,167
after 7 12,136

Both runs passed with Score 100. The saving is the login not happening, not less testing.

The value concentrates where the selection is small, which is to say under --impacted: the tighter the diff, the larger the share of a run that used to belong to a session nobody asked for.

What does not change

A suite that never mentions auth: behaves exactly as it did — the field defaults to true, so most users will never see this. plan still authenticates whenever a recipe is configured, and is pinned by a test that fails if this rule ever leaks into it: it is generating the tests, so it has no selection to consult, and a planner that snapshotted the login wall would draft tests describing the login wall.

No new flag. The information was already in the selection, and asking for it would be asking people to compute what we can read.

--impacted says what the diff did not see

The second fix. --impacted selects from git diff <base>...HEAD, which compares two commits — so uncommitted work was invisible to it. A run against a dirty working tree reported no affected routes, selected nothing, and exited 0 without ever saying what it had not looked at. --fail-on-unmapped could not catch it either, because that guard inspects the changed files the hole had already excluded it from.

The working tree is now read wherever the diff is computed, and every change the diff excluded is named on stderr with the routes it maps to. Two classes stay silent so the warning stays worth reading: a file ignore: already declares irrelevant, and a file already in the diff.

Verified before tagging

The dogfood ran the full suite against a real browser and a real model on the change's branch, before it merged: 8 of 8, Score 100 against a --min-score 80 gate, 95 model calls. The login was performed exactly once for the selections that wanted it, and not at all for those that did not.

Worth recording honestly: the two failure paths this closes — the budget exhausted mid-login, and the unreadable storage_state — are proven by unit tests and by mutation, not by a live run. Neither has a cheap live reproduction, and both were reasoned from the code before they were tested. The mutations are what give that weight: changing some to every turns two tests red, dropping the ceiling's half of the predicate turns two red, and reverting the run's half turns four red, with nothing outside the new block moving.