Accuracy is wrong when ignore_index is set

[dros1986]

🐛 Bug

When the ignore_index argument is set to a value, the MulticlassAccuracy object sets the class accuracy to zero, but considers it in the average calculation.

To Reproduce

import torch
from torchmetrics.classification import MulticlassAccuracy

# simulate the output of a perfect predictor (i.e. preds == target)
target = torch.tensor([0, 1, 2, 0, 1, 2])
preds  = target

metric = MulticlassAccuracy(num_classes=3, average='none', ignore_index=0)
res = metric(preds, target)
print(res)
# it prints [0., 1., 1.]

metric = MulticlassAccuracy(num_classes=3, average='macro', ignore_index=0)
res = metric(preds, target)
print(res)
# it prints 0.6667 instead of 1

Expected behavior

It should not take into account the ignored class.

Environment

• TorchMetrics version: 0.11.4 installed from pip
• Python & PyTorch Version: 3.10.6 - 1.13.0
• Any other relevant information such as OS (e.g., Linux): Ubuntu 22.04

[github-actions]

Hi! thanks for your contribution!, great first issue!

[edumotya]

Actually, the macro accuracy is still wrong when ignore_index is set. It is not covered by the above example (preds == target). The issue is that the FP for the ignore_index class are still taken into account. The metric used to work for older version of torchmetrics, such as 0.9.3.

To Reproduce

import numpy as np
import torch
from torchmetrics.classification import Accuracy

# class 0 -> 100% TP
cm_00 = 100
cm_01 = 0
cm_02 = 0
# class 1 -> 90% TP + 10% class 0
cm_10 = 10
cm_11 = 90
cm_12 = 0
# class 2 -> 100% TP
cm_20 = 0
cm_21 = 0
cm_22 = 100

predictions = np.array(
    (
        cm_00 * [0]
        + cm_01 * [1]
        + cm_02 * [2]
        + cm_10 * [0]
        + cm_11 * [1]
        + cm_12 * [2]
        + cm_20 * [0]
        + cm_21 * [1]
        + cm_22 * [2]
    ),
    dtype=np.int,
)
targets = np.array(
    (
        cm_00 * [0]
        + cm_01 * [0]
        + cm_02 * [0]
        + cm_10 * [1]
        + cm_11 * [1]
        + cm_12 * [1]
        + cm_20 * [2]
        + cm_21 * [2]
        + cm_22 * [2]
    ),
    dtype=np.int,
)

metric = Accuracy(
    task="multiclass",
    num_classes=3,
    average="macro",
    ignore_index=0,
)
res = metric(
    torch.from_numpy(predictions),
    torch.from_numpy(targets),
)
print(res)
# it prints 0.6333 instead of 0.95