Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macro Precision computation is wrong when ignore_index is used #1692

Closed
mpBarbato opened this issue Apr 5, 2023 · 2 comments 路 Fixed by #1821
Closed

Macro Precision computation is wrong when ignore_index is used #1692

mpBarbato opened this issue Apr 5, 2023 · 2 comments 路 Fixed by #1821
Assignees
Labels
bug / fix Something isn't working help wanted Extra attention is needed
Milestone

Comments

@mpBarbato
Copy link

mpBarbato commented Apr 5, 2023

馃悰 Bug

When ignore_index is used in MulticlassPrecision, it zeros out the metric on the class to ignore but it still includes the value in the average computation, leading to wrong results.

To Reproduce

import torch
from torchmetrics.classification import MulticlassPrecision

# simulate the output of a perfect predictor (i.e. preds == target)
target = torch.tensor([0, 1, 2, 0, 1, 2])
preds  = target.clone()

metric = MulticlassPrecision(num_classes=3, average='none', ignore_index=0)
res = metric(preds, target)
print(res)
# it prints [0., 1., 1.]

metric = MulticlassPrecision(num_classes=3, average='macro', ignore_index=0)
res = metric(preds, target)
print(res)
# it prints 0.6667 instead of 1

Expected behavior

It should not consider the ignored class in the average computation.

Environment

  • TorchMetrics version: 0.11.4 installed from pip
  • Python & PyTorch Version: 3.7.9 - 1.11.0+cu113
  • Any other relevant information such as OS (e.g., Linux): Linux Mint 18.3 Sylvia
@mpBarbato mpBarbato added bug / fix Something isn't working help wanted Extra attention is needed labels Apr 5, 2023
@github-actions
Copy link

github-actions bot commented Apr 5, 2023

Hi! thanks for your contribution!, great first issue!

@adamjstewart
Copy link
Contributor

Duplicate of #1691

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working help wanted Extra attention is needed
Projects
None yet
4 participants
@adamjstewart @SkafteNicki @mpBarbato and others