G-mean incorrectly implemented for multi-class

[dabrze]

First of all thank you for this great package!

From what I recon G-mean can be easily used as a multi-class evaluation measure - it is simply the geometric mean of class-wise recalls. Therefore, for binary problems:
G-mean = sqrt(TP/P * TN/N)
or using a different notation
G-mean = (class1_correct/class1 * class2_correct/class2)**(1/2.)

For 3-class problems:
G-mean = (class1_correct/class1 * class2_correct/class2, * class3_correct/class3)**(1/3.)

and so on changing the root/power to 1/4., 1/5., ...

There is no need for micro or macro recalls, in fact they diminish the value of G-mean, which is innately a multi-class measure (rare thing in imbalanced learning).

Sidenote:
One thing you probably know is that G-mean can very easily turn to zero - if one class is completely unrecognized the entire evaluation comes down to zero. With problems involving hundreds of classess thats not a nice thing. Therefore, sometimes people make a "correction" and one class is completely unrecognized instead of multiplying by zero they multiply by some (parmaterized) small value, e.g. 0.001.

[glemaitre]

Thanks. We mess-up on that. We will change this asap.

[dabrze]

Maybe this will help:

import numpy as np
import scipy as sp

from sklearn.utils.multiclass import unique_labels
from sklearn.utils.fixes import bincount
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics.classification import _prf_divide

from unittest import TestCase


def g_mean(y_true, y_pred, labels=None, correction=0.001):
    present_labels = unique_labels(y_true, y_pred)

    if labels is None:
        labels = present_labels
        n_labels = None
    else:
        n_labels = len(labels)
        labels = np.hstack([labels, np.setdiff1d(present_labels, labels,assume_unique=True)])

    le = LabelEncoder()
    le.fit(labels)
    y_true = le.transform(y_true)
    y_pred = le.transform(y_pred)
    sorted_labels = le.classes_

    # labels are now from 0 to len(labels) - 1 -> use bincount
    tp = y_true == y_pred
    tp_bins = y_true[tp]

    if len(tp_bins):
        tp_sum = bincount(tp_bins, weights=None, minlength=len(labels))
    else:
        # Pathological case
        true_sum = tp_sum = np.zeros(len(labels))

    if len(y_true):
        true_sum = bincount(y_true, weights=None, minlength=len(labels))

    # Retain only selected labels
    indices = np.searchsorted(sorted_labels, labels[:n_labels])
    tp_sum = tp_sum[indices]
    true_sum = true_sum[indices]

    recall = _prf_divide(tp_sum, true_sum, "recall", "true", None, "recall")
    recall[recall == 0] = correction

    return sp.stats.mstats.gmean(recall)


class TestEvaluator(TestCase):
    def test_g_mean(self):
        cor = 0.001
        y_true = [0, 0, 0, 0]
        y_pred = [0, 0, 0, 0]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), 1.0, places=10)

        y_true = [0, 0, 0, 0]
        y_pred = [1, 1, 1, 1]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), cor, places=10)

        y_true = [0, 0, 1, 1]
        y_pred = [0, 1, 1, 0]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), 0.5, places=10)

        y_true = [0, 1, 2, 0, 1, 2]
        y_pred = [0, 2, 1, 0, 0, 1]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), (1*cor*cor)**0.33, places=10)

        y_true = [0, 1, 2, 3, 4, 5]
        y_pred = [0, 1, 2, 3, 4, 5]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), 1, places=10)

        y_true = [0, 1, 1, 1, 1, 0]
        y_pred = [0, 0, 1, 1, 1, 1]
        self.assertAlmostEqual(g_mean(y_true, y_pred, correction=cor), (0.5*0.75)**0.5, places=10)

You just have to decide if you want to introduce "corrections" and if so with what defaults (0.0?)

[chkoar]

@dabrze great. PRs are always welcome! In case you want to contribute currently we use nose for testing, so there is no need to inherit from the TestCase

[glemaitre]

@dabrze I agree with @chkoar
You did the job for us on that one. You should put that in a PR.

[dabrze]

@chkoar @glemaitre OK I'll try to prepare a proper PR in the following days

[dabrze]

I'm in a pickle concerning G-mean in conjunction with: 1) IBA, 2) report_imbalanced_multiclass.

1. As far as I know IBA is designed for binary problems and I should probably make an option for G-mean to work as previously on a on-vs-rest basis.
2. I'm not sure what kind of G-mean should be presented in the mutli-class report, which is designed in a one-vs-rest manner. In class rows "-", and g-mean in the avg. row? One vs rest averages as previously? Some other idea? Any suggestions?

[dabrze]

It seems that leaving the output of IBA and the report as they were is the siplest and most reasonable option.

[glemaitre]

I was checking those two metrics and this is not trivial.
The main issue is that they were not though to be multi-class at first.

I have to think more in depth regarding how to associate the output of the dominance
and the g-mean and other metrics for multi-class.