feat: add launch crash benchmark (#334)

* feat: add launch crash benchmark

* fix: remove unused benchmark export

* fix: enforce launch benchmark validity

* fix: harden benchmark evidence audit

* fix: verify benchmark metrics provenance

* fix: publish audited launch benchmark

* fix: reject mixed source capture

* fix: audit all repaired paths

* fix: enforce benchmark identity and portability

* fix: preserve audited run records

* fix: harden benchmark publication

Author: janicduplessis

docs/agent-benchmark-v4.md

@@ -172,6 +172,13 @@ identifiers. Private raw results retain invalid attempts for diagnosis. The
 Markdown report is the machine-readable summary; the viewer is an audit view
 and does not redefine metrics.
 
+The executable coordinator and its machine-local setup contract live in
+[`scripts/agent-benchmark/`](../scripts/agent-benchmark/README.md). The driver
+prepares and commits launch-failure fixtures, dispatches Codex or Claude,
+collects and audits proof, writes `run.json`, cleans owned resources, and
+generates the private report. Credentials, pins, golden build state, and raw
+transcripts remain outside the repository.
+
 ![A benchmark timeline with separate shell and monitored background-process lanes](images/benchmark-background-process.png)
 
 ![A benchmark timeline expanded to 4x with its lane labels pinned during horizontal scrolling](images/benchmark-timeline-zoom.jpg)
@@ -227,6 +234,17 @@ output and `stim logs --errors`; control collects the equivalent Metro and
 simulator logs manually. The injected error text is unique per run so the
 collector can prove that the reported stack and repair refer to this failure.
 
+The coordinator creates a per-run fixture branch, injects and commits the
+exception, and checks out that fixture before dispatch, outside the timed
+interval. The agent starts in the fixture checkout and creates its own isolated
+run worktree from that HEAD, so worktree setup remains part of the measured
+workflow. The agent must launch before inspecting source. An actionable
+diagnosis contains both the run's unique error token and the root-layout source
+location. A valid repair removes that token, relaunches the app, and reaches the
+unchanged Settings proof on the same explicitly targeted simulator. Report
+diagnosis and repair timing separately; do not add crash-suite results to the
+readiness charts.
+
 Run the crash suite only after the four-cell readiness pilot is accepted. Keep
 its goldens, prompts, metrics, and report separate from the normal JS/native
 speed results.

knip.json

@@ -1,5 +1,10 @@
 {
   "$schema": "https://unpkg.com/knip@6/schema.json",
-  "ignore": ["website/**", "packages/stim-cli/shim/**"],
+  "ignore": [
+    "website/**",
+    "packages/stim-cli/shim/**",
+    "scripts/agent-benchmark/stamp.mjs",
+    "scripts/agent-benchmark/watch-app.mjs"
+  ],
   "ignoreBinaries": ["pod"]
 }

package.json

@@ -18,6 +18,7 @@
     "build": "pnpm -r --filter \"./packages/*\" run build",
     "test:e2e": "node --test test/e2e/*.e2e.js",
     "knip": "knip",
+    "benchmark:agent": "node scripts/agent-benchmark/driver.mjs",
     "release:prep": "node scripts/release-prep.mjs",
     "preinstall": "npx only-allow pnpm"
   },

scripts/agent-benchmark/README.md

@@ -0,0 +1,71 @@
+# Agent benchmark driver
+
+This is the executable coordinator used by the published agent benchmarks. It
+keeps machine-local pins, credentials, build artifacts, raw transcripts, and
+device identifiers outside the repository while keeping fixture preparation,
+dispatch, evidence collection, audit, cleanup, and reporting reviewable.
+
+The driver currently runs the iOS readiness and JavaScript launch-failure
+suites described in [`../../docs/agent-benchmark-v4.md`](../../docs/agent-benchmark-v4.md).
+Runs are sequential. Never dispatch two cells against the same benchmark root.
+
+## Machine-local layout
+
+Create a benchmark root outside the repository with these entries:
+
+```text
+benchmark-root/
+  bin/stim
+  golden/
+  pins.env
+  runtime/node_modules/stim-cli/
+  results/
+  state/
+```
+
+`bin/stim` is an executable shim for the pinned `stim-cli` in `runtime`.
+`pins.env` contains the exact fixture, CLI, agent-device, OS, Xcode, Node, and
+CocoaPods values checked by `preflight`; use the keys read by `versionChecks`
+in `driver.mjs`. Keep authentication and raw evidence out of Git.
+
+Set the machine-local paths explicitly:
+
+```bash
+export STIM_BENCH_ROOT=/path/to/benchmark-root
+export STIM_BENCH_FIXTURE=/path/to/clean-trailhead-checkout
+export STIM_BENCH_WORKTREE_PARENT=/path/to/benchmark-worktrees
+export STIM_BENCH_STIM_PACKAGE="$STIM_BENCH_ROOT/runtime/node_modules/stim-cli"
+export STIM_BENCH_CODEX_AUTH=/path/to/codex-auth.json
+export STIM_BENCH_SKILLS_ROOT=/path/to/skills
+```
+
+`STIM_BENCH_CODEX_BIN`, `STIM_BENCH_CLAUDE_BIN`, and
+`STIM_BENCH_AGENT_DEVICE_BIN` can pin non-default executable paths.
+
+## Run a cell
+
+Prepare the golden artifact and parked simulator once, then dispatch, collect,
+and clean one cell:
+
+```bash
+node scripts/agent-benchmark/driver.mjs preflight
+node scripts/agent-benchmark/driver.mjs prepare
+node scripts/agent-benchmark/driver.mjs dispatch gpt-5.6-sol stim launch-crash sol-launch-crash
+node scripts/agent-benchmark/driver.mjs collect /path/to/run-directory
+node scripts/agent-benchmark/driver.mjs cleanup /path/to/run-directory
+node scripts/agent-benchmark/driver.mjs report sol-launch-crash
+```
+
+`dispatch` creates and commits the broken fixture before the timed turn, gives
+the agent the fixture checkout as its starting directory, and requires the
+agent to create the measured run worktree itself. `collect` rejects source
+inspection before launch/error capture, a missing exact repair, a missing
+successful relaunch, a mismatched device, or missing Settings-screen proof.
+
+Run the self-tests before a campaign:
+
+```bash
+node scripts/agent-benchmark/driver.mjs selftest-device-targeting
+node scripts/agent-benchmark/driver.mjs selftest-agent-device-isolation
+node scripts/agent-benchmark/driver.mjs selftest-launch-crash
+```