[errata] The κ bands in the reading list are quoted as if they were a standard #85
Unanswered
akash-coded
asked this question in
Q&A
Replies: 1 comment
Confirmed, and this is a fair hit on our own standard. The derivation page already says "Landis and Koch's bands are conventions, not theorems. Say so Two changes, and one non-change:
Opening this as a docs issue with the thread linked. Good catch — the failure mode here is |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
What the material says
The judge-calibration section presents the Landis–Koch κ bands — slight / fair / moderate /
substantial / almost perfect — as though they were an established threshold, with 0.6 as a pass
mark.
Why I think that is wrong
Landis and Koch (1977) proposed those labels for biomedical inter-rater studies and said
themselves that the divisions were arbitrary. They are a convention, not a standard, and they are
widely criticised for exactly the reason this repository cares about: κ is sensitive to
prevalence, so a fixed band means different things at different class balances.
Quoting 0.6 as a pass mark is the same category of mistake as quoting a recall number without its
corpus — which the README's own third column exists to prevent.
What I think it should say
Report κ with the class balance next to it, use the bands as rough language rather than a
gate, and where a threshold is actually needed, derive it from the decision the judge is feeding
rather than from a 1977 convention.
Where
docs/30-learning/interview-prep/01-mathematical-foundations/cohens-kappa.md, and the notebook 06callout.
All reactions