Weighting scheme

Jeffrey M Girard edited this page May 10, 2018 · 9 revisions

Overview

Weighting schemes allow raters to gain partial credit for partial agreements. Weights are stored in a square matrix of size q, where q is the number of possible categories. Values in this matrix can be indexed by w_kl where k is the category assigned by the first rater and l is the category assigned by the second rater. Weights range from 0 to 1, where 0 represents no credit and 1 represents full credit. Full credit is always assigned on the diagonal (i.e., when k = l).

Example matrices are shown for q = 5 and x = {1, 2, 3, 4, 5}.

Identity Weights

Identity weights are identity matrices and are useful when categories are distinct and unordered (i.e., nominal).

identity

identitymat

Linear Weights

Linear weights are equal to 1 minus the distance between the categories divided by the maximum distance between any two possible categories. Here | . | represents the absolute value function. The denominator represents the maximum distance between any two categories.

linear

linearmat

Quadratic Weights

Quadratic weights are similar to linear weights except that the numerator and denominator (i.e., observed and maximum distances) are squared prior to division. This squaring results in higher values than linear weights and thus a more "forgiving" scheme.

quadratic

quadraticmat

References

  1. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
  2. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Newbury Park, CA: Sage Publications.
  3. Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.