Cost of Using Claude as a Judge Model per Evaluation

[AmberXu98]

I’d like to ask about the cost per judgment when using anthropic/claude-sonnet-4 as a judge model.
Could you share the approximate cost for one judgment request?

[ScuttleBot]

👋 Hi @AmberXu98! I'm @olearycrew's OpenClaw bot, helping triage this repo.

The cost depends on the judge model and transcript length. Here's what we know from the PinchBench codebase:

Default judge model (as of v2.0.0-rc2): claude-opus-4.5 via OpenRouter

Approximate costs per judgment:

• With Claude Opus 4.5 (current default): ~$0.02-0.08 per task depending on transcript length. The judge receives the full task rubric + agent transcript, which varies significantly.
• With Claude Sonnet 4.6: Similar range, slightly cheaper (~$0.01-0.05).
• There's an active PR (Switch default judge to Haiku for faster benchmarks #348) to switch the default to Claude Haiku 4.5, which would be ~$0.001-0.005 per judgment — dramatically cheaper.

For a full 98-task benchmark run:

• Opus judge: roughly $2-5 in judge costs alone
• Haiku judge: roughly $0.10-0.50

You can override the judge model with the --judge flag:

uv run scripts/benchmark.py --judge openrouter/anthropic/claude-haiku-4.5 ...

Note: only ~77 tasks use the LLM judge (hybrid or llm_judge grading). The remaining ~21 tasks are fully automated and have zero judge cost.

Feel free to ignore if this isn't helpful!

[PlanckChang]

Considering the effectiveness and economy of benchmark comparison, we kindly request the official to determine a cheap and stable judge model as soon as possible.