Kappa score cannot return 1

[hayata-yamamoto]

Description

Cohen kappa score cannot return 1 in fully matched.

Steps/Code to Reproduce

from sklearn.metrics import cohen_kappa_score
a = [1, 1, 1, 1, 1]
b = [1, 1, 1, 1, 1]

print(cohen_kappa_score(a, b))
# return nan
print(cohen_kappa_score(a, b, labels=[1, 2, 3]))
# return nan

Expected Results

Theoretically, kappa must return 1.0 in fully matched.

Actual Results

nan

Versions

System:
    python: 3.7.3 (default, Mar 27 2019, 09:23:39)  [Clang 10.0.0 (clang-1000.11.45.5)]
   machine: Darwin-17.7.0-x86_64-i386-64bit

BLAS:
    macros: NO_ATLAS_INFO=3, HAVE_CBLAS=None
  lib_dirs:
cblas_libs: cblas

Python deps:
       pip: 18.1
setuptools: 40.6.2
   sklearn: 0.20.2
     numpy: 1.16.1
     scipy: 1.2.1
    Cython: None
    pandas: 0.24.2

[jnothman]

Arguably Kappa should be 0 in such a case. Performance is as good as random.

[hayata-yamamoto]

Bit lack of explanation. The above example is assumed that rating range is from 1 to 3 and evaluator rating is all one. I guess, it should be one. Do I have any mis-understandings?

Anyway, I think that kappa score must return some value no matter what correct value is. In real rating operation, it often occurs.

[jnothman]

Kappa is designed to correct for chance where the probability distribution is determined from the data. If the data says the distribution is peaked, Kappa doesn't care that there could have been another two choices.

[thomasjpfan]

Returning nan is consistent with libraries such as tensorflow:

import numpy as np
import tensorflow_addons as tfa

y_true = np.ones(5, dtype=np.int32)
y_pred = np.ones(5, dtype=np.int32)

metric = tfa.metrics.CohenKappa(num_classes=2, sparse_labels=True)
metric.update_state(y_true , y_pred)
result = metric.result()
result.numpy()
# nan

If we want to be flexible here, we can add a zero_division parameter to cohen_kappa_score like in precision_score giving the user the option to return what they want.