llm_judge: support non-Anthropic judge routes (LiteLLM / OpenAI-compatible)

[akshaylive]

Summary

llm_judge can only reach Anthropic models. Both judge transports are Anthropic-Messages shaped, LiteLLMRoute is explicitly refused, and resolve_evaluation_route pins evaluation back to a Claude backend even when the agent itself runs through a gateway. There is no configuration under which a non-Claude model can grade.

Current behaviour (main @ 0.11.1)

• criteria/llm_judge.py — _invoke_tool_channel matches BedrockRoute / DirectRoute / LiteLLMRoute; the LiteLLM arm returns "llm_judge: evaluation route must be Bedrock/Direct, got LiteLLM".
• models/routing.py — resolve_evaluation_route: "agent_route is LiteLLMRoute → pin evaluation to a constant Claude backend."
• evaluation/judge_bedrock.py — posts "anthropic_version": "bedrock-2023-05-31" with Anthropic-shaped tools / tool_choice, and the reply is parsed by extract_verdict_from_anthropic_response. So even a non-Claude model hosted on Bedrock cannot be used: to_bedrock_model only vendor-qualifies claude-* ids, and the request body wouldn't be understood regardless.
• agent_judge is documented as "built-in kinds only — agent_judge runs a Claude Code sub-agent."

Why this matters

1. Judge-model choice cannot be evaluated in-product. Any comparison has to be run outside the framework, so the results can't be reproduced by coder-eval itself or wired into CI.
2. Self-preference. The agents under test are frequently Claude, and the grader is always Claude. There is currently no way to run a cross-vendor grader to check whether a model family is scoring its own output favourably — a structural gap in evaluation methodology, independent of cost.
3. Cost. Judging is a small share of total run cost, but the per-token ratio between vendors is large enough to matter as judged volume grows or rubrics get richer.

We ran an offline comparison by replaying recorded judge prompts against a non-Claude model, calling the provider directly outside the framework. It reproduced the incumbent grader's pass/fail verdicts on 84 of 85 criteria. Whether that particular model is the right choice is beside the point — the framework currently makes the question unanswerable without bypassing it.

Proposal

Add an OpenAI-compatible judge transport and allow it to be selected:

1. evaluation/judge_openai.py, alongside judge_bedrock.py / judge_anthropic.py — POST /chat/completions with a forced function tool, same retry/backoff contract as the Bedrock invoker.
2. An OpenAI translation of SUBMIT_VERDICT_ANTHROPIC_TOOL (input_schema → function.parameters).
3. A verdict extractor for choices[].message.tool_calls[], mirroring extract_verdict_from_anthropic_response.
4. A real case LiteLLMRoute() arm in _invoke_tool_channel, and relaxing resolve_evaluation_route so a gateway route can serve the judge instead of being pinned to Claude.
5. Usage mapping (prompt_tokens / completion_tokens → TokenUsage).

Two findings from prototyping all of the above, worth encoding in whatever lands:

• The OpenAI surface rejects reasoning_effort when function tools are present on /chat/completions ("Function tools with reasoning_effort are not supported"); only /responses accepts both. "none" is special-cased and does work on /chat/completions.
• Some reasoning models reject temperature values other than their default ("Unsupported value: 'temperature' does not support 0"), so the judge's temperature=0 cannot be assumed portable across providers.

Related, smaller

(a) claude-sonnet-5 cannot be used as a judge at all. judge_bedrock.py puts temperature in the request body unconditionally, and sonnet-5 responds 400 "temperature is deprecated for this model". Setting model: anthropic.claude-sonnet-5 on an llm_judge criterion therefore fails at dispatch on every criterion. Making the field conditional looks like a one-line fix.

(b) There is no way to set the judge model globally. -D override roots are ('agent', 'run_limits', 'sandbox'), and neither ExperimentDefaults nor ExperimentVariant carries a success_criteria field. Changing the judge model across a suite currently means editing every task YAML that declares an llm_judge, or monkey-patching the default on LLMJudgeCriterion.model (which additionally requires TaskDefinition.model_rebuild(force=True), since the criterion union is compiled at import — patching only LLMJudgeCriterion silently has no effect through the task loader). A success_criteria override root, or a judge section in ExperimentDefaults, would turn judge-model A/B into a config change.

Happy to contribute the transport — we have working implementations of (1)–(3) and (5) from the prototype.

[akshaylive]

Design sketch for (b) — judge model/route as config, without criteria knowing about TaskDefinition

Chatted through a design for the "no way to set the judge model globally" gap (and, while there, the judge-route side of the self-preference concern). Posting it here in case it's useful before anyone starts on the transport work.

Principle: criteria (models/criteria.py, criteria/*.py) stay task-blind — they only ever see their own Pydantic fields plus CheckContext, the orchestrator-resolved bundle already used for route/reference_dir (criteria/base.py). The fix generalizes that existing seam instead of teaching llm_judge to read TaskDefinition.

1. CheckContext gains one generic, namespaced field:

extra: Mapping[str, Mapping[str, Any]] = field(default_factory=dict)
def for_criterion(self, criterion_type: str) -> Mapping[str, Any]: ...

Any checker — built-in or third-party plugin — can read context.for_criterion(self.type_name) without core changes. This also means the judge-model fix isn't a bespoke field; it's the first consumer of a mechanism that composes for future cross-cutting needs.

2. New task-YAML root, checker_context, namespaced by criterion type, layered through the existing 5-layer merge (task → experiment defaults → variant → CLI -D):

checker_context:
  llm_judge:
    model: gpt-5
    route: litellm   # backend SELECTOR, not the route object — no token here

model picks the judge model. route is an ApiBackend string (direct/bedrock/litellm) — a selector, decoupled from the agent's own route, so cross-vendor grading (item 2 in the issue) is expressible without secrets ever touching task YAML.

3. Orchestrator (already holds self.task, at its existing route-resolution call site) resolves the override before building CheckContext:

judge_ctx = task.checker_context.get("llm_judge", {})
eval_route = resolve_evaluation_route(settings, agent_route, backend_override=judge_ctx.get("route"))
self.success_checker = SuccessChecker(self.sandbox, route=eval_route, extra=task.checker_context)

4. resolve_evaluation_route grows one override branch: if backend_override is set, build that specific route from env — same construction resolve_route already does for the agent (LITELLM_BASE_URL/LITELLM_AUTH_TOKEN, AWS_BEARER_TOKEN_BEDROCK/AWS_REGION, ANTHROPIC_API_KEY) — and raise ValueError naming the missing env var if that backend isn't configured, rather than silently falling through to the pin-to-Claude default. Credentials never leave env vars; only the backend choice is task/experiment-authored.

5. llm_judge checker changes by one line:

model = criterion.model or context.for_criterion("llm_judge").get("model") or DEFAULT_JUDGE_MODEL

route is fully consumed upstream — the checker never sees the string "litellm", only the resolved, credentialed ApiRoute.

This is purely the config-plumbing half. It's gated on the actual transport (items 1–3, 5 in the proposal above: judge_openai.py, the tool-schema translation, the verdict extractor, usage mapping) — route: litellm has nothing to select until _invoke_tool_channel has a real LiteLLMRoute arm. Happy to split this into its own PR either before or after the transport, whichever the maintainers prefer.