v0.22.0 — 0.21.0 fixed the form nobody writes

One fix, and the corpus that verified it. It changes verdicts, which is why this is a minor.

0.21.0 fixed the form nobody writes

0.21.0 labelled each redaction with its variable, so a step verifying one secret stopped passing against a page showing another. It was verified with the secret's value written literally in the step, which the mask turns into a label on both sides. People don't write tests that way. They write the placeholder:

- open the Account menu and verify it shows the account email {{env.TEST_OTHER}}

A placeholder isn't a value, so the mask leaves it alone. The judge read {{env.TEST_OTHER}} in the step and [redacted TEST_EMAIL] on the page, and had to work out by itself that those were different secrets. Against a live OWASP Juice Shop, a test built to fail this way passed in 3 of 5 runs. The executor offered a bridge, "[redacted TEST_EMAIL] which is the account email {{env.TEST_OTHER}}", and the judge took it every time its exact input was replayed.

A second path ran beside it. The judgment schema still asked whether the page "satisfies the expectation", the executor's claim, not the step. So when the executor offered "redirected away from login, or an Account menu is accessible", the judge passed on the easy half.

Three changes, because the first two weren't enough

• One vocabulary. The judge now reads every placeholder in the step, the claim and the record as the label the page carries. The executor still sees and types placeholders.
• The right question. The schema asks whether the snapshot shows the step's outcome, and produces its reason before its verdict instead of after.

Measured on the captured inputs, neither half worked alone; together they closed both paths. Live, they halved the rate and stopped there. Given [redacted TEST_OTHER] in the step and [redacted TEST_EMAIL] on the page, in exactly the vocabulary that was meant to settle it, the model still wrote "which matches" in 2 of 4 runs.

• A check on our own tokens. After the model's verdict, a PASS on a step naming [redacted X] is failed when X is neither on the page nor in the step's own actions while another label is on the page. Labels are what the mask writes, so this compares the tool's own output rather than reading prose. It only ever turns PASS into FAIL. The record clause keeps a step that typed a secret into a form it then submitted, and the other-label clause keeps a step asserting that a secret is absent.

On the final code, six live runs: the test built to fail never passed. In one of them the model passed the step twice and the check failed both.

One cost, stated rather than discovered: a step asserting a secret is absent, on a page that shows a different secret, now fails, with a reason naming both labels.

Why there is now a corpus

Asked whether these fixes were going in circles, I counted. Nine changes have touched the verdict since July, each closing a distinct failure found from outside, and none was traced to an earlier fix. The most plausible candidate was tested and ruled out. What was true is that each was verified only against its own reproduction, plus a dogfood suite with none of these cases in it. That's exactly how 0.21.0 missed the placeholder form.

evals/judge/ now holds 33 judge inputs with known verdicts from every incident that produced a wrong one (#31, #35, #87, #112, #120, #121). They're captured from real runs where possible, marked as reconstructed where not, and contain only masked text. npm run eval:judge replays them through the real judge. It needs a key, so it isn't in CI; changing the judge now means running it first. #121's cases are in it, marked as known failures, and the run will say when they stop failing.

Found on the way

Verifying this surfaced three things now filed: a repeated commit that slips past contained-recovery when the model adds a button's text, writing a duplicate (#124); a provider refusal, such as an exhausted balance, failing tests as if the application were broken (#125); and every call reserving 64000 output tokens (#126).