Skip to content

v1.0.8: Failproofai Jev Evaluations

Choose a tag to compare

View all tags
@NiveditJain NiveditJain released this 27 Sep 07:01
· 20 commits to main since this release
v1.0.8
17a025c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Jev comes to failproofai. Regex policies can't tell rm -rf build/ that you asked for from an rm -rf ~ that slipped in. Jev, TypeSafe's classifier, reads each gated tool call against what you actually asked for and judges it above your regex policies. It can use your own key, or run through FailproofAI Cloud on your org's plan with no extra account.

This is the first stable release with Jev. If you don't set Jev up, nothing changes: with no Jev config, every policy decides exactly as it did in 1.0.7.

npm i -g failproofai@latest && failproofai update

Run failproofai update straight after the install so the daemon matches the CLI.

Highlights

Two tiers: a regex floor, and Jev above it

  • Your regex policies stay a hard floor. Jev runs alongside them on each gated call. It can add a deny or an instruction of its own, and it can clear a deny, but only from a policy explicitly marked reviewable.
  • Every policy now has an authority, hard or reviewable, and a reviewedBy list naming the Jev checks that may clear it. Anything unmarked, invalid or alwaysOn is hard, so custom, pack and cloud-managed policies can't be weakened by accident.
  • 15 built-in policies are reviewable. These are the ones that block real work most often: reading outside the workspace, env and secret exposure, destructive SQL, rm -rf, kubectl, terraform, cloud CLIs, force-push and more. sudo, curl | sh and block-work-on-main stay hard on purpose.
  • Measured on 1,332 real tool calls: real work wrongly blocked fell from 33% to 7%, harmful calls blocked rose from 77 to 123 out of 234, and agreement with human labels went from 53% to 71%.
  • Any Jev failure (timeout, rate limit, outage, bad answer) falls back to the regex result, and the reason is recorded. The default Jev budget is 3000 ms per call.

Jev through FailproofAI Cloud, with no second key

  • Create a key with the new machine preset in FailproofAI Cloud (events:add + policies:pull + jev:evaluate), then run:

    failproofai config --token

  • Jev turns on in shadow mode: it logs what it would have done and never blocks. Switch with failproofai jev setup --provider failproofai --mode enforce. Enforce needs an https Cloud URL.

  • Spend comes from your org's plan, with per-org rate limits and a daily cap.

  • The Cloud policy page now shows Jev: a health strip (answered, fallbacks, latency, shadow and enforce machines), a Jev line on the decisions chart, and a "cleared by Jev" column per policy.

  • --no-transcripts never switches Jev on by itself. It stores the key and prints the opt-in command.

Bring your own key

  • failproofai jev setup --provider <typesafe|openrouter|vercel|cloudflare|custom> (or failproofai jev --url --token , which reads the provider from the URL's host).
  • The config lives at ~/.failproofai/jev.json. It's global, written 0600, and refused if it's group- or world-writable.
  • The local dashboard's settings gear has a Jev panel. The key is write-only there: it's never shown back.

New commands

  • failproofai jev status: provider, mode, how many enabled policies Jev may clear, and the last 24 hours (evaluations, fallback rate, latency, what it cleared or would have cleared).
  • failproofai jev test: one real call. It exits non-zero if the answer is too slow or wrong for hooks to use.
  • failproofai jev models: lists the models an endpoint serves and marks the one in use.
  • mode: "off" for any provider keeps the config but stops asking Jev.

Packs can ship Jev checks

  • A pack can carry Jev checks next to its regex policies (semanticPolicies.add({ … })). FailproofAI/jev-policies ships 16 of them, and FailproofAI/policies@2.0.0 carries the reviewable marks.
  • failproofai publish checks a pack's Jev half before it builds. It refuses reserved or built-in check names from outside FailproofAI, packs over the question budget, and alwaysOn. It requires minCliVersion ≥ 1.0.8-beta.0, and writes it for you.
  • policies add and policies show say which of a pack's checks this machine will actually ask.

New built-in policy

  • warn-git-clean: warns before git clean deletes untracked or ignored files (-d, -x, -X). Until now, git clean -fdx was caught by neither tier.

Security

  • Consent can only clear what the user actually named. A clear based on "the user asked for this" now needs every target of a shell command named by the user. It's never given for a command the scanner can't fully read: $VAR, $(…), backticks, heredocs, brace expansion, globs, eval and sh -c. The regex floor stands for those, and Jev's own deny or warning still counts.
  • Pack takeover closed. A pack can no longer supply the Jev check that clears another pack's policies by reusing its name. A byte-identical pack can't make a hard policy reviewable. A release from outside FailproofAI can't install under a FailproofAI/ pack id.
  • Credentials: redaction now covers gateway keys, Anthropic and OpenAI project keys, Authorization values of any scheme, credential flags and private keys. A key in a Jev URL's query string is never shown back, and config --token warns that the key landed in your shell history.
  • protect-env-vars catches more whole-environment dumps: set, export -p, declare -x, and os.environ piped out.

Fixes

  • config --token --url now uses the URL you gave it, and config --token --no-transcripts no longer sends transcripts.
  • Jev no longer judges "inside the project" against wherever the agent last cd'd; the project root is pinned per session.
  • jev setup refuses an endpoint given where a base URL belongs (for example …/v1/models). It also refuses a model that endpoint doesn't serve, and it now shows TypeSafe's own error messages.
  • "Cleared by Jev" counts only clears that actually changed the outcome.
  • failproofaid trusts the OS certificate store, so a self-hosted Cloud behind a private CA works, and its upload and poll errors now name the real cause.
  • failproofai flush counts the batches the daemon actually spools.
  • Repeating --policy, --only, --category or --cli on policies add now applies every occurrence.

Upgrade notes

  • Run failproofai update after the npm install. A CLI and daemon on different protocol versions deny every tool call until they match.
  • No Jev config means no change. 13,400 unconfigured comparisons showed zero differences from the previous evaluator.
  • A pack that declares Jev checks needs failproofai 1.0.8 or later. Older CLIs refuse it and name the remedy.
  • Rolling back: npm i -g failproofai@1.0.7 && failproofai update.

Docs: Jev with your own key · Jev through FailproofAI Cloud · Full changelog: CHANGELOG.md

Assets 8
Loading