Repository navigation
2.4.2
What's Changed
Every llm_as_judge row now describes what it measures.
Shipped as a patch. It changes what published rows contain, so the 1.0.0 policy would call it a minor; it goes out as 2.4.2 as a deliberate exception, alongside 2.4.1's judge-pin fix that it completes. No schema change — the columns already exist and were empty.
Fixed
-
A judge's dimension row (
<grader>.<dimension>) carried no judge information at all: nojudges.*, andjudge_scaleof 0. So0.400on a row could not say who scored it, on what scale, or that the raw number was 2 out of 5 — the scale lived only on<grader>.overall, and recovering the raw point meant joining back to it.Each dimension row now carries the verdict for the dimension it measures, with
judges.pointsscoped to that dimension:metric value scale judges.score judges.points quality.accuracy 0.8 5 [0.8] [{accuracy: 4}] quality.empathy 1.0 5 [1.0] [{empathy: 5}] quality.overall 0.9 5 [0.9] [{empathy: 5, accuracy: 4}]Carrying the full map on a dimension row would put irrelevant numbers on it and invite reading the wrong one.
<grader>.overallis unchanged.This establishes one invariant across every judge row, dimension rows included:
value == mean(judges.score)With one judge they are equal; with a panel,
valueis the mean andjudges.scoreshows the spread — so per-dimension disagreement is readable from the dimension row instead of by unpacking the map on.overall.
Upgrading: notEmpty(judges.name) now matches every judge row rather than only <grader>.overall, so a six-case run goes from 12 judged rows to 48. Anything aggregating over judges must filter to .overall or it multiplies by the dimension count. That is a change in results, not an error.