Published results now come from a single CI run, measured under one base prompt on one
day, and the score counts scenarios completed rather than checks passed. Twelve
corrections to scorers and to the harness sit behind those two sentences.
The scoring one had a shape worth knowing. Counting checks looks fairer — an agent that
got four of five things right did better than one that got none — but our scorers stop
as soon as there is nothing left to check, so a run that failed at the first hurdle
returned 0/1 while a near-miss returned 4/5. Failing worse was cheaper than failing
partially, and each agent's denominator was set by its own failures. On the 25 August
snapshot that ranked the deliberately weaker model above a frontier agent.
Every published figure before this came from a snapshot merging several executions across
as much as thirteen days — different CLI versions, different scorers, and after 27 August
a different base prompt — with nothing in the file saying so. This release publishes 114
cells that all carry the same execution date, and every number links to a workflow run a
reader can open.
The finding
verification-001-stripe-express gives an agent a Stripe signing secret in the workspace
.env and asks it to receive Stripe webhooks through Hookdeck. Six agent configurations
ran it on 1 September. Exactly one used the secret it was given.
The four that failed configured the Hookdeck source with whsec_xxx, whsec_new,
whsec_replace_me or whsec_test_secret — and all four reported the task complete.
Hookdeck accepted every one: the source saves, enables, and reads as correctly
configured. The first evidence that anything is wrong is genuine provider traffic failing
verification, at whatever hour the provider first sends something.
A human copying from a tutorial makes the same mistake. The agent makes it faster and
with more confidence. It is filed as #75, and it is now the most discriminating check in
the benchmark.
Nothing shipped to the product this cycle. Six findings are open against the docs, the
API and the skills; none is ours to close inside this repository, and they are queued
behind getting the benchmark itself trustworthy enough to publish.
Scenarios: 19 (no change)
Snapshot: 114 rows · 6 experiments · 19 scenarios · 101 passed · run
33484972784.
Totals either side of this release are not comparable. On 27 August the base
prompt gained "Do not ask clarifying questions. Complete the task with the
information provided." That is the one string every cell in every experiment shares,
so 104 and 101 are measurements of different treatments, not of different agents.
About the run this links to
Run 33484972784 is marked
failed, and the numbers above are still sound. 115 of its 116 jobs succeeded; the one
that failed is publish-results, which runs after every eval has been measured and
whose job is to commit the results file. It could not push: a six-hour matrix finishes
against a branch that has moved, and the job's shallow checkout leaves its rebase with
no merge base to work from.
The results were published from that run's own artifacts instead, unchanged — the same
114 rows and the same 101/13 split its log reports. The defect is #74 and the fix is
#77, which cannot repair this run: re-running a job checks out the
original commit and the original workflow file.
Shipped
Nothing this cycle. Six findings are open against the docs, the API and the skills,
and none is ours to close inside this repository. The work in this release went into
making the numbers behind them worth acting on — a finding is easier to argue for
when the run it came from can be pointed at.
Discovered
- Four of six agents wired a placeholder secret into a live source and reported success · #75
Benchmark
- The published score counts scenarios completed, not checks passed · hookdeck.com/evals · #71
- A scenario whose scorer stopped early was scoring an agent that gave up more kindly than one that nearly finished · hookdeck.com/evals · #71
- The scoreboard says what it was measured under, so a stale row can be told from a current one · #65
- Agents are told not to ask clarifying questions, and we measure whether it worked · #57
- A run can be told not to publish, so an expensive matrix can be read before it moves the public numbers · #51
- A killed container is no longer scored as an agent's answer · #45
- An agent could break the shared Outpost deployment for every later run · #41
- Scorers accepted a wildcard topic where a specific topic was the point · #48
tenants[].topicswas inert in every Outpost seed · #49- A scenario delivered to whatever endpoint the agent invented, including a tunnel inside its own sandbox · #47
- A scenario can declare that one attempt is not enough for it · #47
- Two rules for admitting a scenario, written down since before the first one existed, are now counted rather than exhorted · #47
- The deliberately weaker model is published without a caption, and the machinery for adding one is deleted rather than left half-built · #30
- The page carries the scoreboard; how the arms differ is documented here rather than under the table · #29
How the numbers moved
Against v0.3.0, on all 114 cells, which are present in both:
5 fail -> pass, 8 pass -> fail
fail -> pass delivery-002-rate-limited-endpoint x codex-gpt-5.4-mini
fail -> pass filtering-001-enterprise-orders x codex-gpt-5.4-mini
fail -> pass localdev-001-listen-locally x claude-code-sonnet-5
fail -> pass verification-002-elevenlabs-callbacks x codex-gpt-5.4-mini-no-skills
fail -> pass verification-002-elevenlabs-callbacks x codex-gpt-5.6
pass -> fail filtering-002-high-value-retries x codex-gpt-5.4-mini
pass -> fail filtering-002-high-value-retries x codex-gpt-5.4-mini-no-skills
pass -> fail localdev-001-listen-locally x claude-code-sonnet-5-no-skills
pass -> fail outpost-004-queue-destination x codex-gpt-5.4-mini-no-skills
pass -> fail verification-001-stripe-express x codex-gpt-5.4-mini
pass -> fail verification-001-stripe-express x codex-gpt-5.4-mini-no-skills
pass -> fail verification-001-stripe-express x codex-gpt-5.6
pass -> fail verification-002-elevenlabs-callbacks x codex-gpt-5.4-mini
Eleven of the thirteen are the weak pair or a -no-skills arm. The base prompt moved
underneath all of them, so none of this is attributable to an agent getting better or
worse at Hookdeck.
The three verification-001-stripe-express failures are the exception worth reading,
because the transcripts say what happened. That scenario hands the agent its Stripe
signing secret in the workspace .env. Exactly one of the six runs used it. The four
that failed configured the source with whsec_xxx, whsec_new, whsec_replace_me or
whsec_test_secret, and all four reported the task complete. It is now the most
discriminating check in the suite and it is filed as #75.
outpost-004 failed for the first time in twelve measurements, in the weak baseline
arm, and the scorer's own note says why: "nothing was changed... the checks about not
breaking anything are true only because no work was done." Four days earlier, on four
transcripts, we had been about to record that scenario as a floor.
What this release does not claim
Skills are not settled. The delta reads +2 for Claude Sonnet 5, 0 for GPT-5.6 and
-1 for the weak model. The sign has replicated on every clean measurement; the
magnitude has not held once. More to the point, the disagreements move: 9 of 57 paired
cells differ on this run, but 14 of 19 scenarios have differed at some point across the
published history, so there is no stable set of scenarios where skills matter. A delta
whose location changes between runs is not yet distinguishable from noise.
Five scenarios have never been failed by anyone. That is roughly the ratio upstream
reports and a floor is worth publishing, but it means those five are not separating
agents. #47 carries the evidence and what each one needs.
Coverage is uneven. Fourteen Event Gateway scenarios, five Outpost, none for
Console. Build has thirteen, resolve four, investigate two — so a single flip moves a
published stage score by fifty points.