v0.1-paper — evalmut: mutation testing for LLM eval graders

Your eval suite passes. Does it actually check anything?

evalmut is mutation testing for LLM eval graders — no LLM-as-judge, fully deterministic. It takes an eval case your grader passes, injects a known defect mined from a documented real-world failure (not invented), and reruns the grader. If the grader still passes a genuinely-wrong output, that's a hole — a class of regression your eval would ship green.

It distinguishes the two ways an eval lies:

• Blind spot — a real defect ships green (a present check that's broken).
• Brittle spot — a correct output gets flagged (a false-positive check that gets quietly recalibrated until it stops catching the real thing).

The honesty invariant: it never infers a hole from a verdict flip. It infers one only from (output-proven-wrong AND grader-passed), where wrongness is established against the case's own ground truth — independently of the grader being tested. Where polarity can't be proven, the operator declines (N/A) rather than guess.

What's in this tag

• 18 mined operators, each naming the concrete documented failure it reproduces (enforced as a test).
• 90 tests pass; every reviewed false-positive class from an 8-round adversarial cold-critique loop is pinned by a regression test.
• Dogfood (pointed at its own dependency gradecore's graders): 91.4% — 3 real holes (1 blind spot, 2 coverage gaps), and it was fair about which was which.
• External port (faithful ports of promptfoo's documented deterministic assertions): 45.5% — 6 holes across contains/is-json/word-count used as correctness gates, with the strong checks (regex, equals) given a clean bill. Not a claim that promptfoo is buggy — a demonstration of which weak-assertion usages inherit a blind spot.
• The paper (paper/evalmut.pdf) documents the method, the honesty guarantee, and the dogfood/external results.

MIT licensed. Built on gradecore, the deterministic no-LLM-judge grading engine.