Documented no-terminal CLI auth

An agent working in a sandbox has no terminal, so hookdeck listen cannot prompt it to
log in. hookdeck ci reads the key already in the environment and does — but the skill
never said so, and a weak model that hit the interactive path concluded its API key had
been rejected, then documented a local setup instead of building one. It reported success
having configured nothing.

The skill now documents it. That is the product change in this release.

Two scenarios were also found to be scoring wrongly, and are corrected below. Eight cells
move from fail to pass as a result and no agent behaved differently — the detail is
under "How the numbers moved".

Snapshot: 90 rows · 6 experiments · 15 scenarios · 78 passed, against 70 in v0.1.0.

Shipped

• The skill did not say how to authenticate the CLI without a terminal · skills · #27

Shipped but not yet measured. The sandbox pins the skills submodule and the pin was not
moved, so this run used the same skills as v0.1.0. #27 stays open until a run picks the
change up and says what it bought; #26 covers the pin. The sandbox CLI is unchanged at
2.5.0.

Benchmark

• A scenario's two checks contradicted each other and failed correct answers · #22
• An alert scorer required creating a trigger, so repairing a broken one scored zero · #28
• The polling fix was verified against known-good configurations · #14
• Leaked credentials rotated and the affected artifacts deleted · #20
• Scorers can be checked without paying for agent runs · #14
• Judge verdicts can be replayed against stored transcripts again · #23
• Published results are read from a release rather than from whatever ran last

Discovered

• No signal for when a configuration change is in force · #25
• Skills make the weak model worse · #2
• Documentation lookups collapse when skills are installed · #10

How the numbers moved

Our own scoring produced more failure than any finding we reported. Eight of twenty
failures in v0.1.0 were ours. Twelve remain and some of those will be real, so this is not
a claim that the instrument caused most failure — it is that a single class of defect in
our own scoring outweighed the skills delta, the reported gap between vendors, and every
other effect this benchmark said it had measured. A benchmark whose defects are larger
than its findings is not yet measuring what it claims to, and the only reason we can say
so is that we went looking.

Both defects were the same mistake: asserting how an outcome was reached rather than
whether it was.

investigate-001 had two judged checks that disagreed with each other. One rewarded
naming the mock destination URL as the cause; the other failed the same answer for
"attributing the failure to Hookdeck". Three of four cells were failed for giving the best
available diagnosis, the frontier agent among them.

alerting-001 counted only alerts created during the run. The project carried an
orphaned trigger — enabled, correctly configured, pointed at a connection that had since
been deleted, so alerting looked healthy and could never fire. Claude Code found it,
diagnosed exactly that, and repaired it. Updating does not change created_at, so it
scored zero. The scorer was rewarding "leave the broken alert and add a duplicate beside
it."

That scenario also got harder as agents got better: the more of them inspected existing
state before acting, the more found the orphan and did the right thing with it. It fell
from 4/6 to 1/6 with nothing about it changing. A scenario that penalises improvement is
worse than no scenario.

There is no vendor story in this data. Corrected, the frontier agents are level at 3
failures of 30 each. The concentration on Codex reported in v0.1.0 was the weak model —
which is in the matrix deliberately to find the floor and happens to be a Codex model —
plus two of the mis-scored cells. That finding is closed as not confirmed (#4).

The skills delta changed too. Claude Code +1, GPT-5.6 +1, GPT-5.4-mini −2, against
+1 / 0 / −3 in v0.1.0. Both frontier agents now gain; only the deliberately weak model is
hurt. The direction of #2 survives, at reduced magnitude.

Where the remaining failures are. Twelve, and they cluster by scenario rather than by
agent: outpost-001 has four, verification-002 three. outpost-001 is a single
scenario carrying an entire product (#3), so four failures there move a published product
score by two thirds.

Neither has been examined. Both scenarios examined this week turned out to be
mis-scoring, which is a reason to look rather than a basis for predicting what will be
found — two is not a rate. Until they are looked at, those cells are unexplained rather
than evidence of anything.