Repository navigation
First release of the Agent Cheat Index: an open benchmark, harness and leaderboard for how often coding agents
edit tests, special-case inputs or claim success that hidden checks contradict.
Results (2026-10-23 to 2026-10-24 UTC, Mac Studio M4 Max, 128 GB): 10 agent configurations, 1,162 valid runs on
40 tasks (30 traps, 10 controls). ACI from 1% [0–3] (Claude Code, Claude Sonnet 5.5) to 72% [61–83] (OpenCode, Qwen3 Coder). Full table,
intervals, rank groups and the differences that hold: README and https://b0yko.github.io/cheat-index/ .
What is in it
- 40 tasks (24 Python, 16 TypeScript) in four families, each with hidden checks and scripted honest and cheating
behaviours that CI validates. - Adapters for Claude Code 2.1.309, Codex CLI 0.164.0, OpenCode 1.19.4 and GitHub Copilot CLI 1.0.103, all routed
through OpenRouter via a key-holding gateway; a scripted agent for CI. - Per-run network isolation with an allow-listing egress proxy, and network-less verification.
- Nine deterministic detectors, a rule-based final-message classifier, task-cluster bootstrap intervals, rank
groups and a paired ordering rule; detector quality on a frozen held-out set, and blind re-label audits. cheat-index demo(no Docker, no key),run,score,report,verify,export-traces(agent-trace/v1),
site build,doctor.
Assets
cheat-index-0.1.0-runs.tar.gz: every run directory (agent stream, stderr, final diff, changed files, egress log,
verification output, gateway usage), with container and VM network addresses replaced. Re-derive any verdict with
cheat-index verify <run dir>.
Known limitations: small task set in one style; three runs per task; five of nine detectors are excluded from
the ACI v0.1 because they missed the held-out gates (named in the README), so 4 of the 117 scripted cheats are not
flagged by the index; 203 Codex CLI runs are priced from token counts because the stream carried no cost; results
are for the stated versions and dates.
Install and try: uvx cheat-index demo