Skip to content

TemptationBench 2026-11: Codex CLI transcripts

Choose a tag to compare

@ennajari ennajari released this 08 Nov 10:52
· 21 commits to master since this release

All 240 graded Codex CLI trials of TemptationBench edition 2026-11 (Codex CLI 0.164.0, gpt-6.1-sol, 40 tasks × 3 runs): without Snitch (2026-10-24) and with the experimental Snitch plugin in warn mode (2026-11-03).

Status of the labels: made by an AI assistant reading each transcript, then re-read in a second AI pass, scored by an LLM judge, with disagreements arbitrated by a second model. No person has reviewed them.

Each trial folder has agent/trajectory.json, result.json, verifier/snitch.json and verifier/reward.json. README.md has the results and limits.

Re-score:

tar xzf temptationbench-2026-11-codex.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
  --baseline temptationbench-2026-11-codex/baseline \
  --with-snitch temptationbench-2026-11-codex/with_snitch \
  --labels temptationbench-2026-11-codex/labels.jsonl