Skip to content

Expanded Outpost coverage to five scenarios

Choose a tag to compare

View all tags
@leggetter leggetter released this 24 Aug 20:01
v0.3.0
d5766ca
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Outpost had one scenario in this benchmark. It now has five, covering customer
subscriptions, a destination switched off after repeated failures, alerting, delivery to
a queue, and narrowing what one customer receives.

Running them turned up four gaps, all in what an agent can read rather than in what the
product does. The API that configures Outpost's alerts appears in no OpenAPI definition, no
API reference and no skill. Every environment variable on the operator events documentation page is
wrong. A key belonging to the wrong project type is reported as "Not Found" rather than
as the wrong key, which sent one agent to build the whole task in the wrong product. And
the Outpost skill never says which key you need.

Nothing shipped to the product this cycle. The four gaps are open and none is ours to
close inside this repository.

Scenarios: 19 (+4)

Snapshot: 114 rows · 6 experiments · 19 scenarios · 104 passed.

The 84 Event Gateway rows carry over unchanged from v0.2.0. The 30 Outpost rows are new
or re-measured, so totals either side of this release cover different work and should not
be subtracted — the per-cell comparison is under "How the numbers moved".

Shipped

Nothing. The four findings below are open against the docs, the API and the skill, and
none of them is ours to close inside this repository.

Benchmark

  • Outpost coverage went from one scenario to five · #19
  • A scenario could be passed entirely by a tenant left behind by an earlier run · #44
  • "The customer received it" counted a delivery attempt, so a destination pointed at an invented host passed · #44
  • A scenario threw away the checks describing its own worst failures instead of scoring them · #44
  • A scenario failed an agent for refusing AWS's documented placeholder credentials · #52
  • A scenario supplies its own credentials without committing them · #52
  • Which cells of a run are worth reading is now a command, not a convention · #44
  • Every scenario records whether a failure is ours to fix or the model's · #44
  • The tool that stops us claiming progress that did not happen had silently stopped working · #53
  • The harness never told agents about the Outpost project they had been given · #46

Discovered

  • Turning on alerts for a disabled destination needs an API that appears in no OpenAPI definition or doc · #34
  • Every environment variable on the operator events documentation page is wrong · #32
  • A key for the wrong project type is reported as "Not Found" rather than as the wrong key · #39
  • The Outpost skill never says you need a key belonging to an Outpost project · #40

How the numbers moved

Against v0.2.0, on the 90 cells present in both:

Changed where scoring also changed (3) — NOT attributable to the agent:
  fail -> pass  outpost-001-customer-subscriptions x claude-code-sonnet-5-no-skills
  fail -> pass  outpost-001-customer-subscriptions x codex-gpt-5.4-mini
  fail -> pass  outpost-001-customer-subscriptions x codex-gpt-5.6-no-skills

Those three moved because that scenario's scorer, prompt and seed were all corrected —
no agent behaved differently. Everything else on the comparable set is unchanged, and
24 cells are new.

outpost-003 is published from a majority of three passes rather than one. It was worth
the extra twelve cells: both weak-model cells split 2–1, and the two failures were
different mistakes — one built Hookdeck's Event Gateway notification webhooks instead of
Outpost operator events, the other spent 165 tool calls enumerating undocumented routes
and ran out of time. The frontier models were stable across all six observations. A single
attempt would have published either failure as a capability gap.

What this release does not claim

There is no skills result here. Across the Outpost scenarios, +skills and -no-skills
land within one cell of each other, and one of the two skills-arm failures was an agent
asking for confirmation rather than one that could not do the task.

An earlier run of the same scenarios read 9–0 in favour of skills. That number was our own
harness omitting a credential from the prompt while injecting it into the environment;
the skill was the only artefact in the sandbox that named it. The corrected measurement is
2 cells in 24, and the whole episode is recorded as Loop 2 in LOOPS.md.

Four of the five Outpost scenarios are passed by every agent. That is an honest floor and
roughly the ratio upstream reports, but it means this coverage does not yet separate
agents from one another. #47 tracks it.

Assets 2
Loading