Skip to content

2.4.2

Choose a tag to compare

@scottpmiller scottpmiller released this 09 Aug 14:46
· 10 commits to main since this release
2.4.2
7098e42

What's Changed

Every llm_as_judge row now describes what it measures.

Shipped as a patch. It changes what published rows contain, so the 1.0.0 policy would call it a minor; it goes out as 2.4.2 as a deliberate exception, alongside 2.4.1's judge-pin fix that it completes. No schema change — the columns already exist and were empty.

Fixed

  • A judge's dimension row (<grader>.<dimension>) carried no judge information at all: no judges.*, and judge_scale of 0. So 0.400 on a row could not say who scored it, on what scale, or that the raw number was 2 out of 5 — the scale lived only on <grader>.overall, and recovering the raw point meant joining back to it.

    Each dimension row now carries the verdict for the dimension it measures, with judges.points scoped to that dimension:

    metric                    value   scale  judges.score  judges.points
    quality.accuracy          0.8     5      [0.8]         [{accuracy: 4}]
    quality.empathy           1.0     5      [1.0]         [{empathy: 5}]
    quality.overall           0.9     5      [0.9]         [{empathy: 5, accuracy: 4}]
    

    Carrying the full map on a dimension row would put irrelevant numbers on it and invite reading the wrong one. <grader>.overall is unchanged.

    This establishes one invariant across every judge row, dimension rows included:

    value == mean(judges.score)
    

    With one judge they are equal; with a panel, value is the mean and judges.score shows the spread — so per-dimension disagreement is readable from the dimension row instead of by unpacking the map on .overall.

Upgrading: notEmpty(judges.name) now matches every judge row rather than only <grader>.overall, so a six-case run goes from 12 judged rows to 48. Anything aggregating over judges must filter to .overall or it multiplies by the dimension count. That is a change in results, not an error.