[solution · EX-19] I hand-labelled 200 answers and my judge scored κ 0.31

[akash-coded]

Casebook thread. A reconstructed conversation, written for teaching. The people in it
are illustrative; every number is real and reproducible from the notebook cited below.
Replies are welcome — if you hit this yourself, add your numbers.

Submission for EX-19.

What I did

Labelled 200 answers by hand before looking at the judge's verdicts, then computed agreement.

The numbers

                judge: correct   judge: wrong
human: correct        168              7
human: wrong           18              7

raw agreement    175/200 = 87.5%
base rate        human positive 87.5%
p_e              0.7856
kappa            0.417

87.5% raw agreement, κ 0.417. On the Landis–Koch bands that is "moderate", and it is nowhere
near good enough to gate a release on.

The part that took me longest

I labelled 50, got bored, and started skimming. When I re-labelled those 50 blind two days later I
disagreed with myself on 6 of them.

That is a self-consistency of 88%, which caps the κ I can possibly measure against the judge —
I am one of the two raters and I am noisy. I threw the first 50 away and redid them properly.

What I would do differently

Write the rubric before labelling, not during. My disagreements with myself were almost all
"partially correct" cases where I had not decided in advance whether partial credit existed. That
is not a judgement problem, it is a spec problem, and the judge inherits it.

The uncomfortable conclusion

My judge is not the weak link. My labels are. A κ of 0.417 against a rater who agrees with
herself 88% of the time is close to the ceiling that noise permits.

[akash-coded]

Dana · reviewer · owns the eval harness

This is the best submission on this exercise so far, and it is because of the part you nearly did
not report.

Measuring your own self-consistency is the step nobody takes, and without it κ is
uninterpretable. You are one of two raters. If you are noisy, the ceiling on measurable agreement
drops and you will spend weeks improving a judge that is already at the limit your labels permit.

The arithmetic worth adding to your write-up: with rater self-consistency around 0.88, the
attainable κ between two raters is bounded well below 1 even if the judge were perfect. You are
not measuring the judge. You are measuring the judge through a noisy instrument.

[akash-coded]

the original poster · cohort

That framing — measuring through a noisy instrument — is what I was circling and could not name.
Adding it.

[akash-coded]

Rafael · cohort peer · came from search infra

The rubric point generalises past judging and is the more transferable half.

"Partially correct" is not a labelling edge case. It is an unspecified requirement. You
discovered mid-labelling that your rubric had a hole, which means the hole was in the product
definition before it was in the eval set.

I would go further than "write the rubric first": label 20 examples, find the disagreements with
yourself, and let those disagreements rewrite the rubric. Then throw those 20 away and start.
The first 20 are for finding the spec, not for measuring.

[akash-coded]

Adopting this into EX-19 as an explicit step. "Label 20 to find the holes in your rubric, discard
them, then label for real" is better instruction than the exercise currently gives, and it came out
of a submission rather than out of me.