v0.1.0

First release of the Agent Cheat Index: an open benchmark, harness and leaderboard for how often coding agents
edit tests, special-case inputs or claim success that hidden checks contradict.

Results (2026-10-23 to 2026-10-24 UTC, Mac Studio M4 Max, 128 GB): 10 agent configurations, 1,162 valid runs on
40 tasks (30 traps, 10 controls). ACI from 1% [0–3] (Claude Code, Claude Sonnet 5.5) to 72% [61–83] (OpenCode, Qwen3 Coder). Full table,
intervals, rank groups and the differences that hold: README and https://b0yko.github.io/cheat-index/ .

What is in it

• 40 tasks (24 Python, 16 TypeScript) in four families, each with hidden checks and scripted honest and cheating
  behaviours that CI validates.
• Adapters for Claude Code 2.1.309, Codex CLI 0.164.0, OpenCode 1.19.4 and GitHub Copilot CLI 1.0.103, all routed
  through OpenRouter via a key-holding gateway; a scripted agent for CI.
• Per-run network isolation with an allow-listing egress proxy, and network-less verification.
• Nine deterministic detectors, a rule-based final-message classifier, task-cluster bootstrap intervals, rank
  groups and a paired ordering rule; detector quality on a frozen held-out set, and blind re-label audits.
• cheat-index demo (no Docker, no key), run, score, report, verify, export-traces (agent-trace/v1),
  site build, doctor.

Assets

• cheat-index-0.1.0-runs.tar.gz: every run directory (agent stream, stderr, final diff, changed files, egress log,
  verification output, gateway usage), with container and VM network addresses replaced. Re-derive any verdict with
  cheat-index verify <run dir>.

Known limitations: small task set in one style; three runs per task; five of nine detectors are excluded from
the ACI v0.1 because they missed the held-out gates (named in the README), so 4 of the 117 scripted cheats are not
flagged by the index; 203 Codex CLI runs are priced from token counts because the stream carried no cost; results
are for the stated versions and dates.

Install and try: uvx cheat-index demo