v0.3.1 — Working evals with Jev

Optional decision-model (Jev) judged-lane for typed eval dimensions.

• Routes binary and closed-label residue (yes/no, choice, ordinal score) to a hosted typed-question decision model, after your deterministic checks run first.
• New stdlib scripts: jev_confidence_route.py, jev_choice_route.py (confidence or top-two-margin routing), jev_cascade_cost.py (batched-question and escalation-tier cost), jev_calibrate.py (temperature scaling fit from your own labels).
• New reference 15-decision-model-judge.md and a preregistration template.
• Opt-in and discovery-only: suggests connecting a key when none is set, warns that a hosted call sends fixture text off your machine, and earns claim-bearing status only after clearing your own kappa-0.8 agreement floor. Nothing changes if you don't use it.

v0.3.0 — Safety/adversarial evals + eval-integrity hardening

Eval-genius can now stress-test whether your AI holds up under attack, and it closes several ways an eval can quietly lie to you. Drop-in upgrade: no breaking changes, same scripts and workflow.

What you can do now

• Test whether your agent resists attack. A new safety-eval branch (14-safety-threat-model.md) walks you from a threat model to a reusable set of attack cases, prompt injection, tool-argument smuggling, data exfiltration, unsafe tool calls, goal hijacks, and grades containment with real code checks (no file left the sandbox, no secret was read), reported per attack type so one weak spot can't hide behind an average.

• Catch an agent that games your grader. When a capable model passes by exploiting the checker instead of doing the task, hidden holdouts, verifier stress-tests, and a "right answer but the trajectory shows a shortcut" check flag it, and an exploitable verifier voids the run instead of banking a fake win.

• Know when there's nothing to measure yet. A new pre-check asks whether an expert could verify the output without redoing the work. If not, the honest result is "can't measure, fix the product first," not a prettier score.

Sharper by default

• Generic metrics can't masquerade as proof — they're allowed to sort and sample, barred from deciding pass/fail.
• Failure categories stay human-owned — you label real traces first; the model only helps cluster after, so it can't flatten your specific bugs into generic buckets.
• Synthetic data earns its place — each item names the real failure it stands in for, and synthetic-only evidence can't back a public accuracy claim.
• Agent evals test the whole session, not one turn — plus an "is the test world even set up right?" pre-flight before the agent runs.
• Skip the eval when a fix is cheaper — a quick triage: fix it now, measure it, or admit it's unmeasurable.
• Reports show every capability family so an overall gain can't hide one that collapsed, and a run cut short by a rate limit is marked unmeasurable instead of counted as a regression.

No API, script, or workflow changes. Deterministic suite (136 tests + 145 subtests) and the full plugin eval suite both green.

Full changelog: v0.2.2...v0.3.0

v0.2.2 — input hardening across all stat scripts

Hardens every utility script so a degenerate or crafted input yields a clear verdict instead of a misleading one. No API or workflow changes; pure robustness.

Fixes

• rate_interval: refuse fractional counts; caution on zero-width intervals; normalize -0.0.
• check_gate: refuse self-comparison (file identity, incl. symlink/hardlink); refuse all-error runs; st_ino==0 realpath fallback.
• judge_agreement: refuse self-comparison; state undefined precision/recall instead of nan; case-insensitive CSV header.
• paired_bootstrap: refuse degenerate resampling and thin evidence; float-coerce deltas so huge scores error cleanly with no traceback.
• hash_fixture: hash the full canonical object (removes duplicate/sibling-key ambiguity), stable same-id tie-break, refuse empty fixtures.
• all scripts: shared duplicate-key rejection on every JSON/JSONL parse; RecursionError guard on deeply-nested input; consistent UTF-8-with-BOM reading.

Tests

23 new tests; suite at 136 passing.

Credits

Includes PR #4 by Dani Kamanovsky (@danikdanik) — paired_bootstrap degenerate-resampling refusal.

Eval Genius v0.2.1

Presentation + discoverability, no behavior change.

• Supported agents shown with real logos. The README now leads with the official Claude, Codex, and Droid marks instead of a text badge.
• Native install in Codex and Droid, documented with copy-paste one-liners (symlink into ~/.codex/skills / ~/.factory/skills). Any other agent still works via SKILL.md.
• Keywords gained codex, droid, cross-agent for discovery.

The skill itself is unchanged: same stdlib-Python, zero-dependency SKILL.md.

Eval Genius v0.2.0

A major expansion: anyone with zero eval knowledge can now go A-to-Z building evals for a product step, a whole AI system, or a benchmark.

New

• Beginner front door: error-analysis workflow that turns real bad outputs into what to measure, plus persona on-ramps and a "unit under test" fork.
• Three full worked walkthroughs (product step / AI system / benchmark) with reproducible toy fixtures.
• Deeper method: customer-feedback evals, judge cost-cascade, partial credit vs binary gates, ablation, dependency mocks/replay, grader hack-resistance, suite lifecycle & graduation, cost ceilings, monitor label loop, transcript-review cadence, inline guardrails, trigger-F1.
• New script: rate_interval.py (Wilson interval for a single rate).
• Self-testing: a 24-case dogfood suite runnable via claude plugin eval; 99 tests including routing-integrity and SKILL.md size caps.
• README + METHODOLOGY revamped around the decision-layer positioning.

Apache-2.0. Install: /plugin marketplace add alexgreensh/eval-genius then /plugin install eval-genius@eval-genius.

Eval Genius v0.1.2

Another small hardening step on the judged-layer gate.

• A judged comparison now requires a pinned judge in both runs. When the items being compared include judged ones, both runs must declare a matching, non-empty judge prompt and model fingerprint. If both arms leave it out, that's CANNOT-MEASURE, not a quiet pass — you can't trust a judged score comparison without proof the same grader produced both. Deterministic-only comparisons are unaffected.

No breaking changes.

Update: /plugin install eval-genius@eval-genius.

Eval Genius v0.1.1

A small hardening pass on top of v0.1.0.

• The gate now fails closed on a one-sided judge fingerprint. If one run records which judge (prompt/model) it used and the other doesn't, that's CANNOT-MEASURE, not a quiet pass. Missing metadata isn't proof the graders matched.
• The run-manifest template now matches exactly what the gate reads. A run you start from the template passes on structure, so only PASS / FAIL / CANNOT-MEASURE reflects the measurement, never the paperwork.

No breaking changes.

Update: /plugin install eval-genius@eval-genius (or add the marketplace first with /plugin marketplace add alexgreensh/eval-genius).

Eval Genius v0.1.0

Eval Genius gives your AI agent the judgment most eval advice skips: when you actually need an eval, how to build one that holds up, and how to read what it tells you, in plain language.

This first release also makes it a proper Claude Code plugin.

Install

/plugin marketplace add alexgreensh/eval-genius
/plugin install eval-genius@eval-genius

Prefer a plain skill folder or another agent? Copy skills/eval-genius/ wherever your agent looks for skills.

What you get

• Tells you if you even need an eval, and which kind fits where you are, from first prototype to production.
• Builds it with you: what to measure, which grader, how many examples, a bar you write down before you look.
• A gate that can't lie: every run ends in PASS, FAIL, or CANNOT-MEASURE, so a crash never gets counted as a pass, and it only compares two runs when they truly measured the same thing the same way.
• Reads the result honestly: against the bar you set, with noise bounds, per-item diffs, and a quick check for a broken gauge before you trust a surprising number.
• Four small, zero-dependency Python scripts you can run on their own: check_gate, paired_bootstrap, judge_agreement, and hash_fixture.
• Reads in five languages: English, 简体中文, Español, 한국어, 日本語.

Works in Claude Code, any agent that loads skills, or straight from your terminal. Apache 2.0.