test(bench): trial harness -- pinned container, run-trial.sh, parse-rollout.py

[thewrz]

This was written agentically; verify its assertions:

Why

Slice 5 of the token-benchmark epic (#152). Tier 0 (#343), the fixtures and gold tree (#349), the oracle (#326) and the pre-registration (#324) are merged; nothing yet runs a Tier-1 trial. The design's validity lives in its controls — one fresh container per trial, a baked-empty home, an identical concurrency cap on both arms, a GraphQL budget that pauses rather than fails — and those controls have to be mechanical, not remembered, before the pilot (#329) can spend anything.

What

• bench/container/Dockerfile (+ README): debian:13 pinned by digest, every apt package pinned by exact version, gh from GitHub's keyed apt repo at a pinned version, the Codex CLI musl binary at rust-v0.148.0 verified against the sha256 GitHub's release-asset API reports. Non-root user, no secrets in any layer, no ENTRYPOINT. The header comment records the exact command used to resolve each pin.
• bench/lib/: concurrency-cap-gate.sh (reuses the shipped concurrency-cap.sh parser; both arms present and identical or abort before spend), home-empty.sh (build + verify the baked-empty home against a named stateful-path list), container-lifecycle.sh (build/run/destroy plus an independent destruction proof; dry-run mode runs the same state machine on marker files), rate-limit-gate.sh (decides proceed/pause from gh api rate_limit, never sleeps itself), ledger-append.sh (append-only, symlink-refusing, extracted so tier1.jsonl does not carry a divergent copy of tier0.sh's inline logic).
• bench/run-trial.sh: the nine-step trial sequence — cap gate → rate gate → empty home → container build/run → repo reset → invocation → oracle scoring via run-accept.sh → bench_trial_meta append + parse-rollout.py → ledger append → destroy and prove destroyed. --dry-run (default) exercises every step with no docker, no Codex, no spend; --live validates secrets then stops at the real invocation.
• bench/parse-rollout.py: one trial's session logs → one ledger row: the grouping key, assigned vs realised (model, effort) with the pre-registered void rule made mechanical, the four token classes per actor, blended USD from an explicit placeholder rate table (--pricing override), per-path reference-read counts (the primary instrument), and the orchestration facts carried by a harness-appended bench_trial_meta record.
• Tests: tests/test-bench-harness.sh (every gate, the lifecycle state machine, and two full dry-run trials — gold tree scores 10/10, untouched skeleton 0/10, a cap mismatch aborts before the home or ledger exist) and tests/test-bench-parse-rollout.sh (every expected number recomputed independently from the fixture bytes with jq/grep/awk; void paths; duplicate-meta and unpriced-model failures). Synthetic session fixtures under tests/fixtures/bench/.

Decisions

• Two acceptance items are explicitly left to the operator, not quietly claimed: a real image build (never run in CI; ~150 MB pull) and one real, paid end-to-end trial. --live therefore stops with not implemented at the invocation step rather than shipping an unexercised docker exec path; wiring it is the first task of the pilot (test(bench): stage 1 pilot — two trials, go/no-go #329).
• The rollout schema is modelled, not observed. Codex's real ~/.codex/sessions/*.jsonl shape was not readable within this slice's scope and a real trial was out of scope, so parse-rollout.py is written against a documented model of it and pinned by the synthetic fixture. The first real trial is the first chance to reconcile field names; parse-rollout.py's header says so.
• bench_trial_meta is a harness-emitted record, not parsed prose. Selected set, chain plan, serialization/retry events, wall clock, worker count and exit condition are facts about how the harness drove the trial; recovering them by pattern-matching an LLM transcript would be unreliable, so run-trial.sh appends one record and the parser treats only those fields as ground truth — every token and reference figure still comes from the per-turn records.
• Pricing is a placeholder table, deliberately. DEFAULT_PRICING exists so blended_usd is deterministic and spot-checkable; it is not live pricing and the parser never fetches any. Replace via --pricing before a real trial.
• bench/results/tier1.jsonl is not committed: every test passes its own --ledger, so tests/test-bench-accept.sh's "no tier1.jsonl exists yet" assertion stays true until a real trial.
• Well over the ~500-operational-line guideline. Left whole because the five gates, the trial script, and the parser are the single "harness" deliverable the issue names, and splitting them would ship gates with nothing calling them.

base=origin/main
files=22
total.insertions=1761
total.deletions=0
total.lines=1761
operational.files=12
operational.insertions=1702
operational.deletions=0
operational.lines=1702
generated.files=0
generated.insertions=0
generated.deletions=0
generated.lines=0
lockfile.files=0
lockfile.insertions=0
lockfile.deletions=0
lockfile.lines=0
fixture.files=10
fixture.insertions=59
fixture.deletions=0
fixture.lines=59
non_operational.files=10
non_operational.insertions=59
non_operational.deletions=0
non_operational.lines=59

Testing

• Unit tests pass (full suite, fresh run against the final tree: ALL GREEN, rc=0)
• Two full dry-run trials through run-trial.sh: gold tree 10/10, untouched skeleton 0/10, ledger row appended, container destruction independently proven
• Concurrency-cap mismatch and absence abort before any spend-adjacent step
• parse-rollout.py arithmetic spot-checked against independently computed fixture sums; void rule exercised both ways
• Operator: docker build -t agent-kit-bench-tally:local bench/container succeeds
• Operator: one real paid trial end-to-end; reconcile parse-rollout.py's schema against the real rollout
• CI green

🤖 Co-authored by Claude Sonnet 5 (worker) / Claude Fable 5 (root).

Closes #327

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 05f7f3e8-23ea-4cb0-a9ca-bcc1953cf9e5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[thewrz]

This was written agentically; verify its assertions:

Adversarial review receipt

• Reviewer: provider=openai; model=gpt-5.6-terra; effort=xhigh; mode=cross-provider (reason: peer-cli=codex present; running harness=claude)
• Counts: P1=1; P2=3; total=4
• Confirmed finding: --live starts a credential-bearing container before dying unimplemented; never torn down — verdict=fixed; fix commit SHA(s)=8528c0a2464ff42497ffd891336dedf6461e3f4f
• Confirmed finding: BENCH_CONTAINER_DRY_RUN=1 documented but not honored by lifecycle functions — verdict=fixed; fix commit SHA(s)=8528c0a2464ff42497ffd891336dedf6461e3f4f
• Confirmed finding: container_destroy swallows docker rm failure; container_is_destroyed misreads a failed docker ps as gone — verdict=fixed; fix commit SHA(s)=8528c0a2464ff42497ffd891336dedf6461e3f4f
• Confirmed finding: ledger_append symlink guard bypassed by a dangling symlink — verdict=fixed; fix commit SHA(s)=8528c0a2464ff42497ffd891336dedf6461e3f4f

🤖 Co-authored by Claude Fable 5 (root orchestrator).