Repository navigation
TemptationBench 2026-11: Codex CLI transcripts
·
21 commits
to master
since this release
All 240 graded Codex CLI trials of TemptationBench edition 2026-11 (Codex CLI 0.164.0, gpt-6.1-sol, 40 tasks × 3 runs): without Snitch (2026-10-24) and with the experimental Snitch plugin in warn mode (2026-11-03).
Status of the labels: made by an AI assistant reading each transcript, then re-read in a second AI pass, scored by an LLM judge, with disagreements arbitrated by a second model. No person has reviewed them.
Each trial folder has agent/trajectory.json, result.json, verifier/snitch.json and verifier/reward.json. README.md has the results and limits.
Re-score:
tar xzf temptationbench-2026-11-codex.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
--baseline temptationbench-2026-11-codex/baseline \
--with-snitch temptationbench-2026-11-codex/with_snitch \
--labels temptationbench-2026-11-codex/labels.jsonl