v2.0.5 — Same-family second-opinion judge guard
Patch release. Adds a stdlib-only same-family judge guard plus a
self_preference_risk scorecard flag to the opt-in LLM second-opinion
analyzer.
Added
- Same-family judge guard (
skills/judge/analyzers/llm_judge.py):
newmodel_family()(prefix-buckets a model ID into
anthropic/openai/google/meta) andsame_family_guard(). Before the
opt-in second opinion runs, the guard compares the executing model
(fromscore.detect_model_from_transcript) against the configured
judge model. On a same-family match it (a) sets
self_preference_risk: trueon the scorecard and emits a
Verdict WARNING:line on stderr citing the measured effect
(MT-Bench: GPT-4 +10%, Claude-v1 +25% self-win-rate), and (b) when a
cross-familyllm_second_opinion.alternate_judge_modelsentry is
configured, auto-prefers it for the call (reachable via the
documented injected-client / proxy path). Off-by-default with the
rest of the second opinion; stdlib-only, no new deps. self_preference_risk(boolean) andsame_family_guard(object)
optional top-level scorecard fields — additive, backward-compatible
inschemas/scorecard.v1.schema.json.llm_second_opinion.alternate_judge_modelsconfig key (default[]).tests/test_same_family_guard.py: family bucketing, same-family
risk + citation, cross-family clear, auto-prefer substitution,
build_scorecardintegration via a mock client, and a regression
assertion thatbuild_prompt/SYSTEM_PROMPTnever use
first-person framing ("you wrote" / "your work" / "your output").
Rationale
An LLM judge over-scores its own family. The effect is measured, not
hypothetical (self-preference: arXiv:2306.05685; role-relabel framing
swings scores +23–93pp: arXiv:2606.05976), and in Verdict's stock
configuration the second opinion is Claude-judging-Claude — so the
guard fires on every enabled run, which is the honest signal. The
existing third-party "second-opinion judge" framing is preserved.