[maths] My judge agrees with me 94% of the time. Why is that not good enough?

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

The question in one line

I hand-labelled 200 answers, ran the judge on the same 200, and they agree on 188 of them. That
is 94%. Notebook 06 then tells me the judge is not usable. Why?

The numbers

judge says correct    181 / 200   (90.5%)
human says correct    179 / 200   (89.5%)
raw agreement         188 / 200   (94.0%)

Where

Notebook 06, judge calibration

[akash-coded]

Marcus · cohort peer

My first read is that 94% is fine and the notebook is being conservative. If I had a junior
analyst who agreed with me on 94% of judgements I would let them work unsupervised.

What is the argument that this is not enough?

[akash-coded]

Priya · cohort peer

The analyst comparison is where it goes wrong, and it took me a while to see why.

You would trust the analyst because you assume they are agreeing for the right reasons. The
94% here is compatible with the judge having no reasons at all — it says "correct" 90.5% of the
time, and if it said "correct" every time it would agree with you on 89.5% of the set.

So most of that 94% was available for free.

[akash-coded]

Dana · reviewer · owns the eval harness

Priya has it, and κ is the arithmetic that makes it precise.

$$ p_e = (0.905)(0.895) + (0.095)(0.105) = 0.810 + 0.010 = 0.820 $$ $$ \kappa = \frac{0.940 - 0.820}{1 - 0.820} = \frac{0.120}{0.180} \approx 0.67 $$

κ ≈ 0.67. That is "substantial" on the Landis–Koch bands and genuinely usable — better than
I expected from the raw number, and better than the notebook's default corpus gives.

The reason the notebook still warns you: κ moves fast in this region. Push the class balance
to 95/5 with the same 94% raw agreement and p_e rises to 0.905, giving κ ≈ 0.37 — the same
judge, the same agreement rate, "fair" instead of "substantial". Your κ is a property of your
label distribution as much as of your judge.

Practical consequence: quote κ with the base rate next to it, always. A κ without its class
balance is not comparable to anyone else's.

[akash-coded]

Marcus · cohort peer

The 95/5 recomputation is the thing that lands it. Same judge, same 94%, and the verdict moves
two whole bands purely because the labels got more lopsided.

[akash-coded]

Marking this because the thread has arrived somewhere better than the question started.

Three things to take from it.

1 · Raw agreement is uninterpretable without the marginals. The full derivation, including
why κ has that particular denominator, is in
Cohen's κ. The short version: 1 − pₑ is the agreement that was
available to be earned beyond chance, so κ is the share of it you actually earned.

2 · κ ≈ 0.67 here is genuinely usable, and I want to be clear the notebook is not saying
otherwise. What it warns about is treating any single κ as a stable property of the judge.

3 · The question that matters more than the number. A judge with moderate but consistent
error can still rank two systems correctly, because a systematic bias applies to both arms and
cancels in the paired comparison. A judge with high but erratic error cannot. So the useful
follow-up is not "is κ high enough" but "is the judge's error correlated with the thing I am
trying to measure?" — which is what the bias probes in judge.py exist to test, and what
EX-20
asks you to extend.

If you are labelling more than 200, or with more than two raters, switch to Krippendorff's α —
it handles missing labels and ordinal rubrics, and degenerates to κ in this simple case.