Skip to content

v0.9.25 — bench trace pack

Choose a tag to compare

@ahwurm ahwurm released this 20 Jul 15:45
· 303 commits to main since this release

Added — localharness bench pack.

Turns a bench results tree into a versioned dataset artifact: {manifest.json, trajectories.jsonl} — one chat-format record per run, labeled with the pass/fail gate verdict. These are graded trajectories (the bench gates judged each one), which makes the pack a regression-eval baseline for evaluating candidate models today, and clean SFT material if fine-tuning is ever justified.

Safety by construction: only files carrying a ScenarioCompleted verdict are packed — live session files are skipped and counted, never packed (they contain real user conversations, which don't belong in a dataset). Any home-path/key/secret pattern in packed content fails the whole build before a single byte is written; there is no silent scrubbing. Prompts are reconstructed from TurnStarted.task_summary, since bench runs drive the agent loop directly and publish no UserMessage event — found and fixed against real run files during verification.

The manifest stamps the harness version, source tree, per-scenario outcome table, and leak-scan status: packs are regenerated per release and supersede cleanly as harness functionality evolves.

Verified end-to-end live: a real 20-run pure_qa bench against the serving model packed to 20 labeled trajectories (20/20 success recorded in the outcome table), 20 sidecar session files correctly skipped, leak scan passed.

Tests: 5 new red→green; full suite 2306 passed / 0 failed.