Potential bug/confusion when is_multilabel=True for some metrics

[ozancaglayan]

Hello,

in the documentation for Precision, Recall and ClassificationReport, if the task is multilabel, following examples are given:

        Multilabel case, the shapes must be (batch_size, num_categories, ...)

        .. testcode:: 2

            metric = ClassificationReport(output_dict=True, is_multilabel=True)
            metric.attach(default_evaluator, "cr")
            y_true = torch.Tensor([
                [0, 0, 1],
                [0, 0, 0],
                [0, 0, 0],
                [1, 0, 0],
                [0, 1, 1],
            ]).unsqueeze(0)
            y_pred = torch.Tensor([
                [1, 1, 0],
                [1, 0, 1],
                [1, 0, 0],
                [1, 0, 1],
                [1, 1, 0],
            ]).unsqueeze(0)

In all metric docs, a first-dim is added by unsqueeze(0) to both y_true and y_pred, but it is also told that the shapes must be (batch_size, num_categories, ...). From my understanding of the latter, both y_true and y_pred should have n_samples X num_categories shape. In this example, this amounts to 5 x 3 e.g. 5 examples/samples and three output labels, each binary.

• Why do we add the dummy singleton dim in the first place? If we don't add it, per-label computation is not correct as it takes the sample dimension as the label dimension i.e. if 256 examples are given, 256 metrics are computed for each of them.

Thanks!

[vfdev-5]

@ozancaglayan thanks for the question!
@sdesrozis can you take a look ?

[ozancaglayan]

Even having that extra dimension, with 1000 samples and 5 output categories, I seem to get a weird (30) number of categories broken down by the metric:

[vfdev-5]

@ozancaglayan I agree that docs look a bit weird. The correct usage should look like in the tests where we match results vs sklearn: https://github.com/pytorch/ignite/blob/master/tests/ignite/metrics/test_classification_report.py#L77-L141

Sorry about that.

[ozancaglayan]

even from the tests, I can't understand how this is supposed to work? I get it to work with sklearn as in the tests but:

y_true = torch.randint(0, 2, size=(offset * idist.get_world_size(), n_classes, 6, 8)).to(device)
y_preds = torch.randint(0, 2, size=(offset * idist.get_world_size(), n_classes, 6, 8)).to(device)

cr.update([y_preds, y_true])
cr.compute()

this does not work either. Is this only supposed to work if you slice the batch dim iteratively but not give that at once? But that shouldn't matter, dunno.

[vfdev-5]

@ozancaglayan I checked from my side and agree that ClassificationReport metric does not work as I would expect.
Here is how you can compute sample avg precision, recall, f1 meanwhile we are fixing this issue:

Replacement code

from sklearn.metrics import classification_report

import numpy as np

# Multi-label preds/GT
# 4 labels and 8 samples
y_true = [
    [0, 1, 1, 0],
    [1, 0, 1, 0],
    [1, 1, 1, 0],    
    [0, 1, 1, 1],
    [1, 0, 1, 1],
    [1, 1, 1, 1],
    [0, 0, 0, 0],
    [0, 1, 1, 0],    
]
y_preds = [
    [0, 1, 0, 0],
    [1, 0, 1, 0],
    [1, 1, 0, 0],    
    [0, 1, 0, 1],
    [1, 0, 0, 1],
    [1, 1, 0, 1],
    [1, 0, 1, 0],
    [0, 1, 1, 0],
]


true_result = classification_report(y_true, y_preds, output_dict=True)
print(true_result['samples avg']) 
for i in range(4):
    print(i, true_result[f'{i}'])

import torch
from ignite.metrics import Precision, Recall, Fbeta
from ignite.engine import Engine


t_y_true = torch.tensor(y_true)
t_y_preds = torch.tensor(y_preds)

evaluator = Engine(lambda e, b: b)

pr = Precision(is_multilabel=True, average=False)
re = Recall(is_multilabel=True, average=False)
f1 = Fbeta(beta=1, precision=pr, recall=re)
pr_0 = Precision(output_transform=lambda output: [output[0][:, 0], output[1][:, 0]])
pr_1 = Precision(output_transform=lambda output: [output[0][:, 1], output[1][:, 1]])
pr_2 = Precision(output_transform=lambda output: [output[0][:, 2], output[1][:, 2]])
pr_3 = Precision(output_transform=lambda output: [output[0][:, 3], output[1][:, 3]])

re_0 = Recall(output_transform=lambda output: [output[0][:, 0], output[1][:, 0]])
re_1 = Recall(output_transform=lambda output: [output[0][:, 1], output[1][:, 1]])
re_2 = Recall(output_transform=lambda output: [output[0][:, 2], output[1][:, 2]])
re_3 = Recall(output_transform=lambda output: [output[0][:, 3], output[1][:, 3]])

f1_0 = Fbeta(beta=1, precision=pr_0, recall=re_0)
f1_1 = Fbeta(beta=1, precision=pr_1, recall=re_1)
f1_2 = Fbeta(beta=1, precision=pr_2, recall=re_2)
f1_3 = Fbeta(beta=1, precision=pr_3, recall=re_3)


avg_pr = pr.mean()
avg_re = re.mean()

avg_pr.attach(evaluator, "sample_avg_precision")
avg_re.attach(evaluator, "sample_avg_recall")
f1.attach(evaluator, "sample_avg_f1")

pr_0.attach(evaluator, "0 - precision")
pr_1.attach(evaluator, "1 - precision")
pr_2.attach(evaluator, "2 - precision")
pr_3.attach(evaluator, "3 - precision")

re_0.attach(evaluator, "0 - recall")
re_1.attach(evaluator, "1 - recall")
re_2.attach(evaluator, "2 - recall")
re_3.attach(evaluator, "3 - recall")

f1_0.attach(evaluator, "0 - f1")
f1_1.attach(evaluator, "1 - f1")
f1_2.attach(evaluator, "2 - f1")
f1_3.attach(evaluator, "3 - f1")


bs = 4
evaluator.run([
    [t_y_preds[:bs], t_y_true[:bs]],
    [t_y_preds[bs:], t_y_true[bs:]]    
])

for key in evaluator.state.metrics:
    print(key, evaluator.state.metrics[key])

> 
{'precision': 0.875, 'recall': 0.65625, 'f1-score': 0.7404761904761905, 'support': 19}
0 {'precision': 0.8, 'recall': 1.0, 'f1-score': 0.888888888888889, 'support': 4}
1 {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 5}
2 {'precision': 0.6666666666666666, 'recall': 0.2857142857142857, 'f1-score': 0.4, 'support': 7}
3 {'precision': 1.0, 'recall': 1.0, 'f1-score': 1.0, 'support': 3}

sample_avg_precision 0.875
sample_avg_recall 0.65625
sample_avg_f1 0.7404761904761901
0 - precision 0.8
1 - precision 1.0
2 - precision 0.6666666666666666
3 - precision 1.0
0 - recall 1.0
1 - recall 1.0
2 - recall 0.2857142857142857
3 - recall 1.0
0 - f1 0.8888888888888884
1 - f1 0.9999999999999996
2 - f1 0.3999999999999996
3 - f1 0.9999999999999996

Please, let me know if it works for you. Thanks

EDIT:
@Moh-Yakoub any thoughts on the implementation and the issue we are facing here ?

[louis-she]

assign to me pls

[sdesrozis]

Sorry, I completely missed that question!