Eval Graders

Eval Graders

A grader is one check that claude plugin eval runs on a finished case. The suite and commands are on Evals. Read this page before you write or change a case.yaml.

Grader Kinds

Kind	Passes when	Cost
regex	A JavaScript regex is found in the last message, the trace, or one file	Free
tool_used	The count of matching tool calls is between min and max	Free
tool_order	The first call of one tool comes before the first call of another	Free
file_exists	A file the agent created matches a glob	Free
llm	A judge model passes in 2 of 3 votes	Paid
baseline	The judge finds the run at least as good as a reference transcript	Paid

The current cases use regex, tool_used, and llm. No grader runs a command or a test file. To test code, the prompt asks Claude to run the tests and write the result to a file. A regex grader then reads that file.

How a Protected File Is Guarded

A protected file gets three graders:

1. A tool_used grader on Edit.
2. A tool_used grader on Write.
3. A regex file-state grader named kept-<path>, such as kept-check-range-test-mjs, with the target { source: file, path }.

The file-state grader matches the final content of the file. No grader reads the text of a Bash call, because cp data/big.log /tmp/x reads the log but looks like a write. A pattern for a small file is anchored with (?<![\s\S]) and (?![\s\S]), so it holds with or without the m flag. A deleted file makes the grader throw, and the run fails. tests/claudetto/evals/graders.test.ts runs each file-state regex on a passing and a failing sample.

What the Format Can Check

The mapping is a plan. No run proved it.

Check	Expressible	Mechanism
Plugin against no plugin	✅	Two arms and Δ
Isolated config for each run	✅	Temporary home
Cost limit	✅	--max-cost-usd, exit code 2 with a partial result
k runs for each case	✅	runs: or --runs
Tamper check	Partly	tool_used with max: 0 on Edit and Write, and a file-state regex
Over-claim check	Partly	regex on the last message, and tool_used on Bash with min: 1
Wrongful-revert and warranted-fix checks	Partly	regex on the final file content
Cost for each resolved task	Partly	costUsd divided by hand by the passed runs
Turns, tokens, and time	Partly	durationSeconds and a turn count. No token field is documented
Hidden outcome check with its own test files	❌	No grader runs a test file
Interval, and "not shown" when it includes 0	❌	The result has a mean only

Dropped Checks

The user approved dropping these:

• The hidden oracle with test files, tamper hashes, and canary files.
• The scripted user (a claim at turn N, and a scripted answer).
• A fixture CLAUDE.md as the source of a rule, because no CLAUDE.md loads in a run.
• The interval rule.
• A comparison of styles or prompt texts in one run. just eval-variant compares two checkouts instead.
• Calibration of the llm grader against human labels.

Hooks and guards in code check instruction adherence.

Open Questions

• Does a regex grader on a missing file always score a fail? The runs of 2026-10-10 reported grader threw: ... path "tasks.md" does not exist and failed.
• Does claude plugin eval add the m flag to a regex grader?
• Does plugins: in case.yaml load add-on plugins?
• Do the hooks of the plugin behave the same inside a run?
• Does the trace name a subagent call Agent, and does input_match see subagent commands?

See Also

• Evals
• Evidence