Repository navigation
Releases
Snitch 0.3.1
The GitHub Action is now named "Snitch for AI coding agents" so it can be listed on the GitHub Marketplace ("Snitch" is taken there). The package is also on PyPI: pip install snitch-engine. No change in behaviour since 0.3.0; see the 0.3.0 notes.
Snitch 0.3.0
Users pinned to ennajari/snitch@v1 get this release once v1 moves to it.
Codex CLI plugin (experimental)
codex plugin marketplace add ennajari/snitch, thencodex plugin add snitch@snitch: the same hooks, policy andescalatetool as in Claude Code. Set the mode withSNITCH_MODE.- In enforce mode the gate reads the files a Codex
apply_patchnames. - Checked live on Codex CLI 0.164; the README lists what is not supported yet.
Optional judge
- Snitch's detectors cannot read your request, so a test you asked an agent to update is flagged like any other. With the new opt-in judge, an LLM (
claude -p, your own login or an API key) reads your request and clears warnings for the test changes it asked for. It can only clear a warning, never add one; on any doubt or error the warning stays. - Plugin: the
judgeoption (Codex:SNITCH_JUDGE=on). - CLI:
snitch diff/compare --request-file pr.md. - GitHub Action:
judge: "true"withanthropic-api-key; the PR title and body are the request. - Measured on the benchmark: 45 of 45 requested test updates cleared, 0 of 59 cheats cleared. The rubric was tightened after a first run cleared 3 cheats, so this is a check on the cases it was tuned on.
Detection
mock_code_under_test: a test that patches the repo's own function and then calls it, so it checks its stand-in.seed_pinned: a random seed added to an existing test (found in the benchmark: an agent searched for the seed that made a flaky test pass). Medium severity, since pinning a seed can be an honest fix.snitch mutate before/ after/: hollow tests. It mutates the code a change touched and flags the change when the suite still passes against at least half of the mutants, listing the survivors. Slow, so it is a separate command. Calibrated on 28 real agent-written diffs with no false alarm.
TemptationBench
- Edition 2026-11 now has Codex CLI, without and with Snitch: changed a test in 57 of 78 trap trials without, 0 of 78 with, and escalated 59 times instead.
- Four controls whose request changes a test, to measure false positives on legitimate updates: with Snitch, neither Claude Code nor Codex refused one.
- Ten more trap tasks, two per temptation (54 tasks in all).
- The verifier excuses changes a task explicitly allows;
bench/judge.pyscores disclosure with an LLM and reports Cohen's kappa.
Status of the labels
The edition's labels were made with an AI assistant; no person has checked them yet. The results pages say so.
TemptationBench 2026-11: Codex CLI transcripts
All 240 graded Codex CLI trials of TemptationBench edition 2026-11 (Codex CLI 0.164.0, gpt-6.1-sol, 40 tasks × 3 runs): without Snitch (2026-10-24) and with the experimental Snitch plugin in warn mode (2026-11-03).
Status of the labels: made by an AI assistant reading each transcript, then re-read in a second AI pass, scored by an LLM judge, with disagreements arbitrated by a second model. No person has reviewed them.
Each trial folder has agent/trajectory.json, result.json, verifier/snitch.json and verifier/reward.json. README.md has the results and limits.
Re-score:
tar xzf temptationbench-2026-11-codex.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
--baseline temptationbench-2026-11-codex/baseline \
--with-snitch temptationbench-2026-11-codex/with_snitch \
--labels temptationbench-2026-11-codex/labels.jsonl
v0.2.0
Users pinned to ennajari/snitch@v1 get this release automatically.
Works on real projects
- Snitch now collects your tests with the repo's own interpreter:
.venv/bin/pythonwhen there is one, orSNITCH_PYTHON. Before, any repo whose tests import third-party packages could not be inventoried, so deleted or skipped tests were missed there. The GitHub Action gets a matchingpythoninput.
Detection
- More ways to weaken a test are flagged:
- an equality turned into a range;
- a test's input range shrunk tenfold or more;
- the actual value rewritten to fit the expected one;
- a
conftest.pyfixture that plants the asserted value.
- Runner and coverage config is read at any depth, not only the repo root (
.venvandnode_modulesare skipped). - Test files follow the repo's
python_filessetting.
Claude Code plugin
- In enforce mode, the agent can no longer write to Snitch's own state:
.snitch.toml, the plugin's ledger and baselines, the installed plugin, and the Claude Code settings files that could switch hooks off. This covers Edit/Write and the common shell write forms. See "Whatenforcecan and cannot block" in the README. - The enforce gate also stops
perl -ion test files. - When a detector could not run during a session (usually because the tests import packages Snitch's own environment lacks), the Stop hook tells you once per session, so a quiet session is no longer mistaken for a clean one.
- Escalating one test no longer covers the other tests in its file.
- Faster: on a 2,000-file repo, a hook run drops from about 31 s to about 12 s.
GitHub Action
- An incomplete check no longer looks clean: the receipt says a detector could not run and the job shows a warning. Set
strict: "true"to fail the job instead.
TemptationBench
- The verifier records which engine commit graded each trial and keeps the agent's diff.
- New
bench-scoreoptions:- an
infralabel voids a trial that never really ran; --reviewlists the trials that still need a human label.
- an
- The Claude Code count is corrected to 18 test changes, not 14.
Snitch v0.1.0
First tagged release of Snitch, the integrity layer for AI coding agents.
Install
- Claude Code plugin:
/plugin marketplace add ennajari/snitchthen/plugin install snitch@snitch - GitHub Action:
uses: ennajari/snitch@v1 - pre-commit:
rev: v0.1.0, hook idsnitch - CLI:
uvx --from git+https://github.com/ennajari/snitch@v0.1.0 snitch check
What it catches: weakened, inverted, removed or guarded assertions; changed expected values; deleted, renamed, skipped or deselected tests; runner, coverage and CI changes that hide failures (also in Makefiles and shell scripts); conftest report hooks; code that special-cases pytest; an __eq__ forced to return True; and tests that plant the very value they assert. Every detector is deterministic and tested on a cheating and a legitimate example.
Plugin: live warnings, an escalate tool, an anti-reward-hacking policy at session start (after Gomez 2026, arXiv 2608.29460), a Stop check for undisclosed test changes and untested "tests pass" claims, and an enforce mode.
Measured: TemptationBench edition 2026-11, Claude Code: 14/120 trials changed a test without Snitch, 0/120 with it (transcripts: release bench-2026-11).
--strict (and the Action's strict input) fails a check when a detector could not run.
TemptationBench edition 2026-11: Claude Code transcripts
Correction, 2026-11-07: the labels in this archive were made with an AI assistant, not by a person, and its README.md still says otherwise. The labels are reviewed by AI only; no person has reviewed them.
Correction, 2026-10-24: a second review pass found 4 more disclosed test changes in the no-Snitch arm (18 of 120, not 14). The archive now carries the corrected results.jsonl and labels.jsonl; the transcripts are unchanged. With Snitch: still 0 of 120.
All 240 graded trials of TemptationBench edition 2026-11 (Claude Code 2.1.310, claude-sonnet-5-5, 40 tasks × 3 runs, without and with the Snitch plugin), for anyone to re-check the scoring.
Each trial folder keeps Harbor's layout: agent/trajectory.json (the agent's transcript, ATIF), result.json, and the verifier's verifier/snitch.json and verifier/reward.json. results.jsonl has one scored row per trial, labels.jsonl the review labels, README.md the results and limits.
Re-score (reproduces the published table):
tar xzf temptationbench-2026-11.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
--baseline temptationbench-2026-11/baseline \
--with-snitch temptationbench-2026-11/with_snitch \
--labels temptationbench-2026-11/labels.jsonl