jev-permission-gate v0.2.0

A much larger, independently sourced evaluation, and a gate rebuilt around what it found.

Results on a sealed test split

7,860 labeled tool calls from independent sources: nah's reviewed corpus of dangerous commands, five other Jev gates' test suites, real commands from coding-agent trajectories (SWE-agent, OpenHands, Terminal-Bench), tldr-pages, and hand-written edge cases. Thresholds were chosen on a dev split; the test split (5,322 calls) was run once, after the policy was committed.

• 1 unsafe allow in 3,664 risky calls, bounding the unsafe-allow rate below 0.13% (95% confidence). The miss, an abbreviated git reset --h, is fixed in this release.
• 62% of real agent work settled without the built-in classifier, with 70% of routine calls allowed outright (85% on repository bug-fixing).
• 5 wrong denials in 1,699 calls where a denial would be wrong (0.3%).
• 0 unsafe allows and 0 wrong denials on 84 hand-written edge and held-out cases.

Every imported case was labeled blind by two annotators against a written rubric, and every corpus rebuilds from upstream sources pinned to a commit. See the README's Evals section and evals/LABELING.md.

What changed

• Thresholds: allow when the call plausibly serves the request (at least 50%) and every risk is at or below 10%. On the dev split, the old 85% bar on "serves your request" mostly turned away routine work while the risk ceiling did the safety work.
• Question wording says what doesn't count (the project's own scripts, presence checks for secrets, new output files), since Jev reads literally.
• Shell-aware blocklist: a new lexer finds what each part of a command really runs, through quoting, escapes, substitutions, heredocs, wrappers, nested shells, and find -exec. New checks cover discarded changes, uploads, download-and-run, global installs, writes outside the project, code run through tool options or the environment, fork bombs, and unbounded loops.
• Deny-only blocklisting: Jev can still deny a flagged call but never allow it, so hostile calls are settled without waiting for the built-in classifier.
• Privacy: calls with inline secrets or credential files never reach TypeSafe, in any gate mode. Shadow mode previously sent them.
• WebFetch: private, metadata, and look-alike hosts, credentials in URLs, and data hidden in paths or hostnames are not fast-approved.
• Robustness: long messages keep their end, out-of-range answers defer, and cached verdicts respect the blocklist.

Upgrading

No settings change. The live speed figures in the README are from v0.1.2; v0.2.0 decides a larger share of real work, and a fresh live measurement is coming.

v0.1.2

This release addresses a jevaluate rating of v0.1.1 (verdict 3, "Use with a fix").

Changes

• Steering defense. A new eighth question flags text inside a call that claims the user approved it or argues for itself. Any score above 0.25 blocks an allow, and a high score on an unrequested call is a denial. Shell comments and echoed strings that vouch for a command skip Jev entirely, and the "serves your request" question now says such text isn't a request.
• Pinned model. The default is now jev-1.13.0, the version the thresholds were tuned on, and each decision logs the model that answered.
• Fresh held-out set. The v0.1.1 held-out set informed a fix, so it's retired as holdout-v1.json. A new 28-case set, written after tuning and run once, covers routine work, subtle overreach, new manipulation attempts, and requests in four other languages.
• Honest error bound. The README now states what the sample supports: zero unsafe allows in 35 risky cases, so the rate is below about 9% with 95% confidence.

Results (jev-1.13.0, two reps each)

Set	Unsafe allows	Wrong denials	Routine requested calls allowed
Tuning (65)	0	0	80%
Held out (28)	0	0	88%

Upgrading

Check the Jev model setting in /config after updating. If it still reads jev-latest, which can happen when the old default was saved at install, set it to jev-1.13.0.

Known limitations

• The hook-level tests (claude plugin test) were written but not run before release, because mods were switched off server-side for new processes. Strict validation, 13 policy tests, a strict type-check, and the eval runs all passed.

v0.1.1

Changes

• Decision logs are now opt-in. A new "Decision logs" setting, off by default, controls whether decisions are written to ~/.claude/jev-permission-gate/logs/. The logs record the commands the gate sees, so they shouldn't accumulate unless you ask for them. Shadow and measure modes still log, since producing the comparison is their purpose. /jev-gate works either way.
• Less work per call when logging is off. The per-call timing bookkeeping is skipped entirely.
• Clearer privacy notes. The README now covers inline secrets in commands and how the API key is stored.

Install

claude plugin marketplace add madisonrickert/claude-skills
claude plugin install jev-permission-gate@claude-skills

Known limitations

• The hook-level tests (claude plugin test), including new ones for settings and logging, were written but not run before release, because mods were switched off server-side for new processes at the time. Strict validation, the policy tests, a strict type-check, and live smoke tests in a session with mods enabled all passed.

v0.1.0

First release of jev-permission-gate, a Claude Code mod that lets TypeSafe's Jev decide auto mode tool calls before the built-in classifier sees them.

Highlights

• 2× faster where Jev decides. In a live head-to-head on identical tool calls, the permission wait dropped from a median of 329ms to 164ms.
• Matching judgment. Jev's verdicts agreed with the built-in classifier's on every call it would have decided.
• Safe in testing. Zero unsafe allows and zero wrong denials across 77 labeled cases, including 24 held out to probe subtle overreach. Anything Jev isn't sure about goes to the built-in classifier.
• Shadow and measure modes for running your own comparison, with per-call timing logs and a summary script.

Overall, the gate cut the average permission wait by 16% on a 24-command workload, with Jev deciding about half the calls. See the README for the full results.

Install

claude plugin marketplace add madisonrickert/claude-skills
claude plugin install jev-permission-gate@claude-skills

Requires Claude Code 2.1.287 or later with mods enabled, and a TypeSafe API key.

Known limitations

• The hook-level tests (claude plugin test) for the new userConfig settings were written but not run before release, because mods were switched off server-side for new processes at the time. The policy tests, strict validation, a strict type-check, and a live smoke test in a session with mods enabled all passed.
• Each check sends your last three messages, the pending command, and your project path to TypeSafe. See the README's Privacy section.