Skip to content

2.4.1

Choose a tag to compare

@scottpmiller scottpmiller released this 09 Aug 14:27
· 11 commits to main since this release
2.4.1
1953c75

What's Changed

The judge's identity is now in its provenance pin.

Shipped as a patch. Both changes below alter a published value, so the 1.0.0 policy would call this a minor at least; it goes out as 2.4.1 as a deliberate exception, because the old pin was answering a provenance question wrongly and the sooner it stops the fewer runs are affected.

Fixed

  • Scorecard.judge_version now includes the judge's model: anthropic:claude-sonnet-4-6@v1 rather than judge@v1.

    It was key@judge_version. Swapping the judge's model while leaving the declared judge_version alone produced a byte-identical pin, so any provenance check reading that field passed a comparison against a baseline scored by a different model:

    before the swap: judge@v1
    after  the swap: judge@v1     ← same pin, different judge
    

    Now:

    before the swap: anthropic:claude-sonnet-4-6@v1
    after  the swap: anthropic:claude-opus-4-1@v1
    

    The model is the thing most likely to change and the thing a declared version is most likely to miss.

Changed

  • A single judge's key defaults to its provider (anthropic) instead of the literal judge. A panel already defaulted to key or provider, so the one-judge case was the odd one out, and judge named nothing the grader column did not already say. It appears in the judges.name column.

Upgrading: every suite with a judge re-baselines once, because judge_version is part of what identifies a comparable run — the intended behaviour, just paid all at once. Queries filtering judges.name = 'judge' need the provider instead. Set key: on a judges: entry to pin a name of your own.