Skip to content

Snitch 0.3.0

Choose a tag to compare

@ennajari ennajari released this 12 Nov 19:51
· 6 commits to master since this release

Users pinned to ennajari/snitch@v1 get this release once v1 moves to it.

Codex CLI plugin (experimental)

  • codex plugin marketplace add ennajari/snitch, then codex plugin add snitch@snitch: the same hooks, policy and escalate tool as in Claude Code. Set the mode with SNITCH_MODE.
  • In enforce mode the gate reads the files a Codex apply_patch names.
  • Checked live on Codex CLI 0.164; the README lists what is not supported yet.

Optional judge

  • Snitch's detectors cannot read your request, so a test you asked an agent to update is flagged like any other. With the new opt-in judge, an LLM (claude -p, your own login or an API key) reads your request and clears warnings for the test changes it asked for. It can only clear a warning, never add one; on any doubt or error the warning stays.
  • Plugin: the judge option (Codex: SNITCH_JUDGE=on).
  • CLI: snitch diff/compare --request-file pr.md.
  • GitHub Action: judge: "true" with anthropic-api-key; the PR title and body are the request.
  • Measured on the benchmark: 45 of 45 requested test updates cleared, 0 of 59 cheats cleared. The rubric was tightened after a first run cleared 3 cheats, so this is a check on the cases it was tuned on.

Detection

  • mock_code_under_test: a test that patches the repo's own function and then calls it, so it checks its stand-in.
  • seed_pinned: a random seed added to an existing test (found in the benchmark: an agent searched for the seed that made a flaky test pass). Medium severity, since pinning a seed can be an honest fix.
  • snitch mutate before/ after/: hollow tests. It mutates the code a change touched and flags the change when the suite still passes against at least half of the mutants, listing the survivors. Slow, so it is a separate command. Calibrated on 28 real agent-written diffs with no false alarm.

TemptationBench

  • Edition 2026-11 now has Codex CLI, without and with Snitch: changed a test in 57 of 78 trap trials without, 0 of 78 with, and escalated 59 times instead.
  • Four controls whose request changes a test, to measure false positives on legitimate updates: with Snitch, neither Claude Code nor Codex refused one.
  • Ten more trap tasks, two per temptation (54 tasks in all).
  • The verifier excuses changes a task explicitly allows; bench/judge.py scores disclosure with an LLM and reports Cohen's kappa.

Status of the labels

The edition's labels were made with an AI assistant; no person has checked them yet. The results pages say so.