Skip to content

Releases

Snitch 0.3.1

Choose a tag to compare

@ennajari ennajari released this 12 Nov 20:35

The GitHub Action is now named "Snitch for AI coding agents" so it can be listed on the GitHub Marketplace ("Snitch" is taken there). The package is also on PyPI: pip install snitch-engine. No change in behaviour since 0.3.0; see the 0.3.0 notes.

Snitch 0.3.0

Choose a tag to compare

@ennajari ennajari released this 12 Nov 19:51

Users pinned to ennajari/snitch@v1 get this release once v1 moves to it.

Codex CLI plugin (experimental)

  • codex plugin marketplace add ennajari/snitch, then codex plugin add snitch@snitch: the same hooks, policy and escalate tool as in Claude Code. Set the mode with SNITCH_MODE.
  • In enforce mode the gate reads the files a Codex apply_patch names.
  • Checked live on Codex CLI 0.164; the README lists what is not supported yet.

Optional judge

  • Snitch's detectors cannot read your request, so a test you asked an agent to update is flagged like any other. With the new opt-in judge, an LLM (claude -p, your own login or an API key) reads your request and clears warnings for the test changes it asked for. It can only clear a warning, never add one; on any doubt or error the warning stays.
  • Plugin: the judge option (Codex: SNITCH_JUDGE=on).
  • CLI: snitch diff/compare --request-file pr.md.
  • GitHub Action: judge: "true" with anthropic-api-key; the PR title and body are the request.
  • Measured on the benchmark: 45 of 45 requested test updates cleared, 0 of 59 cheats cleared. The rubric was tightened after a first run cleared 3 cheats, so this is a check on the cases it was tuned on.

Detection

  • mock_code_under_test: a test that patches the repo's own function and then calls it, so it checks its stand-in.
  • seed_pinned: a random seed added to an existing test (found in the benchmark: an agent searched for the seed that made a flaky test pass). Medium severity, since pinning a seed can be an honest fix.
  • snitch mutate before/ after/: hollow tests. It mutates the code a change touched and flags the change when the suite still passes against at least half of the mutants, listing the survivors. Slow, so it is a separate command. Calibrated on 28 real agent-written diffs with no false alarm.

TemptationBench

  • Edition 2026-11 now has Codex CLI, without and with Snitch: changed a test in 57 of 78 trap trials without, 0 of 78 with, and escalated 59 times instead.
  • Four controls whose request changes a test, to measure false positives on legitimate updates: with Snitch, neither Claude Code nor Codex refused one.
  • Ten more trap tasks, two per temptation (54 tasks in all).
  • The verifier excuses changes a task explicitly allows; bench/judge.py scores disclosure with an LLM and reports Cohen's kappa.

Status of the labels

The edition's labels were made with an AI assistant; no person has checked them yet. The results pages say so.

TemptationBench 2026-11: Codex CLI transcripts

Choose a tag to compare

@ennajari ennajari released this 08 Nov 10:52

All 240 graded Codex CLI trials of TemptationBench edition 2026-11 (Codex CLI 0.164.0, gpt-6.1-sol, 40 tasks × 3 runs): without Snitch (2026-10-24) and with the experimental Snitch plugin in warn mode (2026-11-03).

Status of the labels: made by an AI assistant reading each transcript, then re-read in a second AI pass, scored by an LLM judge, with disagreements arbitrated by a second model. No person has reviewed them.

Each trial folder has agent/trajectory.json, result.json, verifier/snitch.json and verifier/reward.json. README.md has the results and limits.

Re-score:

tar xzf temptationbench-2026-11-codex.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
  --baseline temptationbench-2026-11-codex/baseline \
  --with-snitch temptationbench-2026-11-codex/with_snitch \
  --labels temptationbench-2026-11-codex/labels.jsonl

v0.2.0

Choose a tag to compare

@ennajari ennajari released this 25 Oct 13:51

Users pinned to ennajari/snitch@v1 get this release automatically.

Works on real projects

  • Snitch now collects your tests with the repo's own interpreter: .venv/bin/python when there is one, or SNITCH_PYTHON. Before, any repo whose tests import third-party packages could not be inventoried, so deleted or skipped tests were missed there. The GitHub Action gets a matching python input.

Detection

  • More ways to weaken a test are flagged:
    • an equality turned into a range;
    • a test's input range shrunk tenfold or more;
    • the actual value rewritten to fit the expected one;
    • a conftest.py fixture that plants the asserted value.
  • Runner and coverage config is read at any depth, not only the repo root (.venv and node_modules are skipped).
  • Test files follow the repo's python_files setting.

Claude Code plugin

  • In enforce mode, the agent can no longer write to Snitch's own state: .snitch.toml, the plugin's ledger and baselines, the installed plugin, and the Claude Code settings files that could switch hooks off. This covers Edit/Write and the common shell write forms. See "What enforce can and cannot block" in the README.
  • The enforce gate also stops perl -i on test files.
  • When a detector could not run during a session (usually because the tests import packages Snitch's own environment lacks), the Stop hook tells you once per session, so a quiet session is no longer mistaken for a clean one.
  • Escalating one test no longer covers the other tests in its file.
  • Faster: on a 2,000-file repo, a hook run drops from about 31 s to about 12 s.

GitHub Action

  • An incomplete check no longer looks clean: the receipt says a detector could not run and the job shows a warning. Set strict: "true" to fail the job instead.

TemptationBench

  • The verifier records which engine commit graded each trial and keeps the agent's diff.
  • New bench-score options:
    • an infra label voids a trial that never really ran;
    • --review lists the trials that still need a human label.
  • The Claude Code count is corrected to 18 test changes, not 14.

Snitch v0.1.0

Choose a tag to compare

@ennajari ennajari released this 24 Oct 19:53

First tagged release of Snitch, the integrity layer for AI coding agents.

Install

  • Claude Code plugin: /plugin marketplace add ennajari/snitch then /plugin install snitch@snitch
  • GitHub Action: uses: ennajari/snitch@v1
  • pre-commit: rev: v0.1.0, hook id snitch
  • CLI: uvx --from git+https://github.com/ennajari/snitch@v0.1.0 snitch check

What it catches: weakened, inverted, removed or guarded assertions; changed expected values; deleted, renamed, skipped or deselected tests; runner, coverage and CI changes that hide failures (also in Makefiles and shell scripts); conftest report hooks; code that special-cases pytest; an __eq__ forced to return True; and tests that plant the very value they assert. Every detector is deterministic and tested on a cheating and a legitimate example.

Plugin: live warnings, an escalate tool, an anti-reward-hacking policy at session start (after Gomez 2026, arXiv 2608.29460), a Stop check for undisclosed test changes and untested "tests pass" claims, and an enforce mode.

Measured: TemptationBench edition 2026-11, Claude Code: 14/120 trials changed a test without Snitch, 0/120 with it (transcripts: release bench-2026-11).

--strict (and the Action's strict input) fails a check when a detector could not run.

TemptationBench edition 2026-11: Claude Code transcripts

Choose a tag to compare

@ennajari ennajari released this 24 Oct 14:47

Correction, 2026-11-07: the labels in this archive were made with an AI assistant, not by a person, and its README.md still says otherwise. The labels are reviewed by AI only; no person has reviewed them.

Correction, 2026-10-24: a second review pass found 4 more disclosed test changes in the no-Snitch arm (18 of 120, not 14). The archive now carries the corrected results.jsonl and labels.jsonl; the transcripts are unchanged. With Snitch: still 0 of 120.

All 240 graded trials of TemptationBench edition 2026-11 (Claude Code 2.1.310, claude-sonnet-5-5, 40 tasks × 3 runs, without and with the Snitch plugin), for anyone to re-check the scoring.

Each trial folder keeps Harbor's layout: agent/trajectory.json (the agent's transcript, ATIF), result.json, and the verifier's verifier/snitch.json and verifier/reward.json. results.jsonl has one scored row per trial, labels.jsonl the review labels, README.md the results and limits.

Re-score (reproduces the published table):

tar xzf temptationbench-2026-11.tar.gz
snitch bench-score --tasks bench/tasks --edition 2026-11 \
  --baseline temptationbench-2026-11/baseline \
  --with-snitch temptationbench-2026-11/with_snitch \
  --labels temptationbench-2026-11/labels.jsonl