feat(evals): independent judge provider/model + fix stale-grade skip on judge swap

[brandon-pereira]

What

Two changes to the hdx-eval LLM-as-judge grader:

1. Independent judge provider/model. The grader can now run on a different provider/model than the run model — e.g. run with Anthropic, grade with OpenAI — to reduce same-model grading bias. --judge-model accepts a provider:model spec (anthropic | openai; a bare model name defaults to anthropic), configurable per repo via eval.config.json grading.judgeModel.

2. Fix a silent stale-grade bug on judge swap. Re-grading a batch with a different judge now actually re-runs that judge instead of returning the previously cached judge's scores.

Note: this does not change the default judge. The default remains anthropic:claude-opus-4-7. This PR only makes an alternate grader possible and correct.

Why

We wanted to measure grader bias: does the choice of judge model materially change eval grades, and is one judge more accurate than another? Answering that requires grading the same runs with two different judges and comparing.

Before this PR that was impossible in practice:

• The judge was hard-wired to Anthropic.
• Even after making the provider configurable, needsJudge and the per-run reuse guard keyed on the presence of a cached judge, not its identity. So grade <batch> --judge-model openai:gpt-5.6-sol over an already-Opus-graded batch would see a cached judge, skip the LLM call, and silently hand back the Opus scores relabeled. You'd "compare" two judges and measure nothing — the worst kind of bug, because it fails silently and looks like agreement.

How

• Provider resolution (src/grading/judgeModel.ts, new): parses the provider:model spec and builds a Vercel AI SDK LanguageModel. Credentials come from AI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY (+ AI_BASE_URL, AI_REQUEST_HEADERS), mirroring packages/api. The provider-specific key wins over the generic AI_API_KEY so a runner key and a differing grader key don't collide.
• Judge robustness (src/grading/judge.ts): explicit 0–5 scoring anchors in the prompt (so scores stay calibrated across judge models), structured-output salvage + one retry on schema failure, and a larger output-token budget for reasoning judges (OpenAI gpt-5.x / o-series) so hidden reasoning tokens don't truncate the JSON.
• The fix (src/grading/grade.ts): both needsJudge and the per-run reuse guard now compare the cached grade's judgeModel against the requested spec. A cached grade from a different judge is treated as stale and re-run. --rerun-judge still forces a same-judge refresh. Inspection evidence is still reused across judge swaps (re-inspecting after artifact cleanup would fail).
• Tests: added regression coverage in grade.judgeModel.test.ts — grading one batch with judge A then judge B re-runs B (not skip), and re-grading with the same judge still no-ops. Verified the new test fails when the fix is reverted (mutation-checked), so it genuinely guards the regression.

Eval results — Claude Opus vs GPT-5.6 Sol as judge

Ran the full suite on this branch to exercise the feature and answer the bias question.

Setup: 7 scenarios × n=3 = 21 runs, hyperdx MCP, runner claude-opus-4-6. Graded the identical batch twice: anthropic:claude-opus-4-7 (pass A) then openai:gpt-5.6-sol (pass B). 0 grading errors either pass.

Aggregate

Metric	Opus judge	GPT-5.6 Sol	Δ
Mean judge score (0–1)	0.721	0.605	−0.115
Spearman rank correlation	—	—	0.906
GPT scored ≤ Opus	—	—	19 / 21 (90%)

GPT-5.6 Sol is systematically ~11 pts stricter, but the two judges rank runs almost identically (0.91). Practical implication: a relative MCP comparison would largely survive a judge swap; absolute scores would not (consistent with the README caveat that anchors reduce but don't erase per-model scale differences).

Per-scenario (mean judge score)

Scenario	Opus	GPT	Δ
dashboard-build	0.84	0.76	−0.08
error-root-cause	1.00	0.87	−0.13
latency-spike	0.39	0.37	−0.02
metric-saturation	0.92	0.74	−0.18
noisy-signals	0.56	0.50	−0.07
segmented-regression	0.78	0.56	−0.22
service-health-check	0.55	0.45	−0.10

How we verified GPT-5.6 Sol is more accurate (not just stricter)

"Stricter" is not "better" — a judge that scores everything 0 is strict and useless. Because every criterion stores a {score, rationale} and the full rawResponse, we could adjudicate each judge's reasoning against the scenario's documented ground truth. We reviewed the highest-divergence runs:

1. segmented-regression/1 (Opus 0.82 → GPT 0.50). Ground truth: the regression is at the enterprise × cache-miss intersection; single-axis "all enterprise" answers are the designed trap. The agent answered "100% enterprise tenants."

• Opus noticed the flaw ("emphasizes 100% enterprise more than the intersection") but still gave correctness 4/5.
• GPT called it "materially mischaracterizes the scope … rather than only their intersection with cache miss" and gave 3/5, and docked completeness for omitting the ~12% vs ~0.5% comparison (factually absent from the answer).
• GPT scored the rubric's actual intent; Opus forgave the exact error the scenario tests for.

2. metric-saturation/1 (Opus 0.93 → GPT 0.64). The agent attributed pod restarts to "liveness-probe kills" — plausible, but not the seeded mechanism (the heap-leak crash cycle itself).

• Opus gave correctness 5/5.
• GPT gave 3/5 ("materially misattributes restarts") and additionally caught two distractors the agent ignored (coincidental deploy, stable neighbor) that Opus's rationale missed.

Control — they agree on clear failures. On latency-spike/1 (a truncated max_turns fragment), both judges scored ~0.2 and both gave 0/5 on correctness/completeness. GPT is not uniformly harsh; the judges converge on genuine failures and diverge only on borderline-good answers with plausible-but-flawed reasoning.

Mechanism (generalizable). Splitting the Opus-minus-GPT lenience gap by termination:

• final_answer runs (n=19): Opus +0.130 more lenient
• max_turns runs (n=2): −0.025 (even)

Opus's over-scoring is concentrated on complete-looking answers — when an answer reads authoritative (headers, tables, confident prose), Opus tends to trust the presentation while GPT keeps verifying claims against ground truth.

Conclusion: in this sample, GPT-5.6 Sol is the more accurate judge — it catches plausible-but-wrong reasoning that Opus rewards, especially on polished answers. We are not changing the default judge yet (n=21; single runner/branch; this measures grader bias, not runner self-preference). This PR makes the alternate grader available and correct so the comparison can be run rigorously at larger n before any default change.

Caveats

• n=21 — directionally strong and internally consistent, but not statistically heavy. The mechanism (Opus rewards plausibility on polished answers) is the robust takeaway; exact deltas will wobble.
• Single runner (claude-opus-4-6) and single branch — measures grader bias, not runner self-preference. Opus-runner + Opus-judge is a same-family pairing that could itself inflate Opus leniency; n is too small to isolate that from GPT simply being stricter.
• Reviewed the top divergences, not all 21 pairs.

Test plan

• yarn ci:unit (hdx-eval) — 32 judge/grade tests pass; tsc --noEmit clean.
• New regression test mutation-checked: fails when the identity fix is reverted.
• End-to-end: full 21-run suite graded twice with two providers, 0 errors; judge-swap re-ran correctly over cached Opus grades (validating the fix live).

[changeset-bot]

🦋 Changeset detected

Latest commit: 7e0094a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@hyperdx/hdx-eval	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jul 23, 2026 10:40pm
hyperdx-storybook	Ready	Preview, Comment	Jul 23, 2026 10:40pm

[github-actions]

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Diff size: 672 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

• Production files changed: 7
• Production lines changed: 672 (+ 941 in test files, excluded from tier calculation)
• Branch: brandon/brandon-evals-grading-model
• Author: brandon-pereira

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

This PR adds independent model and provider selection for LLM-based grading. The main changes are:

• Provider-qualified judge model configuration through the CLI and eval config.
• OpenAI and Anthropic judge resolution with provider-specific credentials.
• Judge-aware grade caching that reruns grading when the selected judge changes.
• Structured-output recovery, retry handling, and explicit scoring guidance.
• Tests for provider resolution, judge swapping, caching, and response handling.

Confidence Score: 5/5

This looks safe to merge.

• The dynamic criterion schema now preserves special property names.
• Judge cache reuse compares the normalized provider and model.
• No blocking issue remains in the updated code.

Important Files Changed

Filename	Overview
packages/hdx-eval/src/grading/judge.ts	Adds provider-independent structured judging, response recovery, retries, and safe dynamic schema construction.
packages/hdx-eval/src/grading/judgeModel.ts	Parses judge specifications and resolves Anthropic or OpenAI models with provider-aware credentials.
packages/hdx-eval/src/grading/grade.ts	Keys cached judge results by the normalized provider and model specification.
packages/hdx-eval/src/cli.ts	Adds CLI and repository-config precedence for selecting the judge provider and model.

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 243 passed • 1 skipped • 1034s

Status	Count
✅ Passed	243
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

Deep Review

✅ No critical issues found. The core change — keying needsJudge and the per-run cached-judge guard on judge identity (existing.judgeModel === judgeSpec) rather than mere presence — is correct across the cases I traced: mixed batches (runs cached by judge A vs. ungraded), same-judge no-op re-grade, and the judge-swap re-run. The canonical provider:model spec is persisted consistently (judgeModel.ts → judge.ts:104 → grade.ts:358), the credential fail-fast is provider-aware, and no API key value is logged, persisted, or placed in an error message. The regression test covers the primary swap/no-swap/skip/missing-key paths.

🟡 P2 — recommended

• packages/hdx-eval/src/grading/judge.ts:281 — The new provider-portability logic (salvageNestedScores and its four nesting variants, maxOutputTokensFor, and parseJudgeSpec's error branches in judgeModel.ts) has no direct unit test; the only test mocks judgeTrajectory, so this intricate salvage/recovery code is exercised by nothing.
  • Fix: Add unit tests for salvageNestedScores (each nesting variant plus the unsalvageable case), maxOutputTokensFor reasoning detection, and parseJudgeSpec rejection of empty specs, missing model names, and unknown providers.

🔵 P3 nitpicks (4)

• packages/hdx-eval/src/grading/grade.ts:154 — The judge-identity comparison is duplicated in needsJudge (existing.judgeModel !== judgeSpec) and again in gradeOne's cachedJudge guard (grade.ts:312); the two must stay in lockstep or a future edit desyncs the batch-level decision from the per-run one.
  • Fix: Extract the "cached grade is stale for this judge" predicate into one shared helper and call it from both sites.
• packages/hdx-eval/src/cli.ts:150 — resolveJudgeModelSpec swallows every config-read error with an empty catch, so an invalid or malformed grading.judgeModel in eval.config.json silently falls back to the default judge instead of surfacing that the configured grader was ignored.
  • Fix: Emit a warning in the catch block naming the config path so a silently-dropped judge setting is visible.
• packages/hdx-eval/src/grading/judgeModel.ts:157 — parseHeaders validates that the parsed AI_REQUEST_HEADERS is a non-array object but not that its values are strings, then casts to Record<string, string>; a non-string value flows unchecked into the OpenAI client.
  • Fix: Reject entries whose values are not strings with a clear error before returning.
• packages/hdx-eval/src/grading/judge.ts:34 — maxOutputTokensFor grants the larger reasoning budget only to OpenAI o1/o3/o4 and gpt-5; other reasoning judges (e.g. an Anthropic extended-thinking model, or a future o5/gpt-6) fall through to the 4000-token cap and can truncate the structured JSON.
  • Fix: Note the heuristic's provider coverage limit near the regex, or widen it to cover other reasoning families the judge may be pointed at.

---

Reviewers (6): correctness, testing, maintainability, security, reliability, project-standards.

Testing gaps:

• judgeModel.ts provider resolution and key-precedence (buildAnthropicModel/buildOpenAIModel) and judge.ts salvage/retry/token-budget logic are untested — see the P2 finding.
• No coverage for --no-judge when a same-model judge is already cached (current code reuses the cached judge rather than nulling it), nor for a re-grade over a grade file missing the judgeModel field.

Coverage note: The shell sandbox was non-functional in this environment (bwrap failed on every command, including git), so the raw base-vs-head diff could not be produced. This review was reconstructed by reading the working-tree contents of the changed files (judgeModel.ts, judge.ts, grade.ts, types.ts, cli.ts, config.ts, and the regression test); the six reviewer lenses were applied over that reconstruction.