Precision Metric: must have at least one example before it can be computed

[erksch]

🐛 Bug description

When using the precision metric and the model output does not predict any positives (neither true or false), then Ignite throws an error:

ignite.exceptions.NotComputableError: Precision must have at least one example before it can be computed.

I guess this is expected because you can not compute the precision without any positives. But you would usually just apply an epsilon to the denominator and it is computable and you even do that in the code. But then I don't understand why to throw an error when positives are empty in the first place?

Maybe I am wrong here, but my intuition is that this check can just be removed.

Environment

• PyTorch Version: 1.8.1
• Ignite Version: 0.4.4
• OS: Linux
• How you installed Ignite: pip
• Python version: 3.9

[vfdev-5]

@erksch thanks for the report!
Precision implementation handles different options like binary, multiclass and multilabel and the check

ignite/ignite/metrics/precision.py

Line 43 in 711a991

if is_scalar and self._positives == 0:

ensures that update method was called at least once. By the way, I think the error is not raised for multilabel case which is inconsistent.

When using the precision metric and the model output does not predict any positives (neither true or false),

This is unclear, actually. Which case do you have: binary, multiclass, multilabel ?

For example, multiclass case, let's assume the predictions to be same for all classes and the ground truth is only 0-class

from ignite.metrics import Precision

pr = Precision()

bs = 32
nc = 5

pr.update((torch.ones(bs, nc), torch.zeros(bs, dtype=torch.long)))

print(pr._true_positives)
print(pr._positives)
print(pr.compute())
> tensor([32.,  0.,  0.,  0.,  0.], dtype=torch.float64)
> tensor([32.,  0.,  0.,  0.,  0.], dtype=torch.float64)
> tensor([1., 0., 0., 0., 0.], dtype=torch.float64)

[erksch]

Thank you for your quick response!

I am using the Precision metric in a label wise metric (multiclass classification)

def get_single_label_output_fn(class_idx):
  def wrapper(output):
    y_pred, y = output
    y_pred = F.one_hot(y_pred.argmax(dim=1).long(), num_classes)
    y = F.one_hot(y, num_classes)
    return y_pred[:, class_idx], y[:, class_idx]
  return wrapper

for class_idx in classes:
  precision = Precision(average=False, output_transform=get_single_label_output_fn(idx))
  # ... recall & f1

Together with create_supervised_evaluator.

My intuition for label wise precision is to have one hot encoded y_pred and y and then just look at the given class.

---

I wonder a little about the documentation because it states:

y must be in the following shape (batch_size, …).

But at the same time

In binary and multilabel cases, the elements of y and y_pred should have 0 or 1 values

If the shape of y is not (batch_size, num_categories), which should be one_hot, then I need to use indices, but that contradicts the second note.

Edit: Oh sorry I guess I am mixing up multilabel and multiclass right now. I guess I need to use indices for y, I'll get back to you if I fixed that up :)

[erksch]

About your example, I wonder why y_pred is a torch.ones(bs, nc)? Shouldn't it be one_hot encoded?

[vfdev-5]

@erksch I agree that our docs is not very clear about how things are working (sorry about that, we are trying to improve that).

In multiclass case, the metric is inteneded to work on logits coming from the model and raw labels coming the dataloader:

1. scalar case

• Logits tensor y_pred.shape = (batch_size, num_classes), e.g. y_pred ~ [[-10.1, 0.2, 10.3, 2.1], ...] for 4 classes
• Targets y_true.shape = (batch_size, ), e.g. y_true ~ [0, 1, 3, 2, ...]

2. vector/image case

• Logits tensor y_pred.shape = (batch_size, num_classes, H, W), e.g. image segmentation with pixel-wise classification probabilities
• Targets y_true.shape = (batch_size, H, W), e.g. image segmentation with pixel-wise classes.

Hope this helps.

[erksch]

Thanks!

Ah yes I see in the Precision source that you perform one_hot on y_true and argmax + one_hot on y_pred.
But for label-wise metrics you need to use one_hot targets in output_transform so you can slice them for the class you are looking for (see my example above). But then the Precision metric applies processing on the targets that does not really fit anymore and gives wrong results.

[erksch]

Ah nevermind it does work :D Because precision thinks the input is a binary classification and does not apply the tranformations of multiclass labes.

[vfdev-5]

Actually, Precision with average=False already gives the precision per class:

print(pr.compute())
> tensor([1., 0., 0., 0., 0.], dtype=torch.float64)

But then the Precision metric applies processing on the targets that does not really fit anymore and gives wrong results.

If you would like to handle your data as binary case: y_pred ~ (batch_size, ), e.g. y_pred ~ [0.5, 0.2, ...] and y_true ~ (batch_size), e.g. y_true ~ [0, 1, 0, 1, ...], you have to provide binarized values for y_pred ...
See here:

• https://pytorch.org/ignite/metrics.html#ignite.metrics.Precision

def thresholded_output_transform(output):
    y_pred, y = output
    y_pred = torch.round(y_pred)
    return y_pred, y

precision = Precision(output_transform=thresholded_output_transform)

[erksch]

Actually, Precision with average=False already gives the precision per class:

Just tried that and it works great and is much simpler! Thank you very much for taking the time with me!

Just for completion, here is a script that reproduces the initial error:

import torch
import torch.nn.functional as F
from ignite.metrics import Precision

batch_size = 10
num_classes = 5

def get_single_label_output_fn(class_idx):
    def wrapper(output):
        _y_pred, _y = output
        return _y_pred[:, class_idx], _y[:, class_idx]
    return wrapper

prs = []

for idx in range(num_classes):
    pr = Precision(average=False, output_transform=get_single_label_output_fn(idx))
    prs.append(pr)

# Random true and pred labels
# y_pred = torch.FloatTensor(batch_size).uniform_(0, num_classes).floor().long()
# y = torch.FloatTensor(batch_size).uniform_(0, num_classes).floor().long()

y_pred = torch.ones(batch_size).long()
y = torch.zeros(batch_size).long()

y_pred_one_hot = F.one_hot(y_pred, num_classes)
y_one_hot = F.one_hot(y, num_classes)

print("y pred:", y_pred_one_hot)
print("y:", y_one_hot)

for i, pr in enumerate(prs):
    transform = get_single_label_output_fn(i)
    pr.update(transform((y_pred_one_hot, y_one_hot)))
    print("class:", i, "precision:", pr.compute().item())

But if this kind of stuff is not really intended then we may leave everything at it is and refer to your method for handling label wise metrics.

[vfdev-5]

But if this kind of stuff is not really intended then we may leave everything at it is and refer to your method for handling label wise metrics.

I think the code from #1991 (comment) is simplier :)

In addition, you can do "metrics arithmetics" to fetch specific class precision as a its own metric:

from ignite.metrics import Precision

torch.manual_seed(12)

pr = Precision()

# Metrics arithmetics:
# use __getitem__ operator and .item() directly
pr_2 = pr[2].item()
pr_4 = pr[4].item()

bs = 32
nc = 5

pr.update((torch.rand(bs, nc), torch.randint(0, nc, size=(bs, ))))

print(pr_2.compute(), pr_4.compute())
> 0.375 0.16666666666666666

And everything work smoothly with Engine.

[erksch]

Just to clarify for me, Precision with average=true gives the Macro Precision and not the Average (Micro) Precision? Because we average the precisions of each individual class instead of actually counting total positives and negatives. But I guess if you know that it is not really a big deal because Average Accurcy = Micro Precision = Micro Recall = Micro F1

[vfdev-5]

Yes, Precision with average=True is macro precision (same as precision_score(..., average="macro")).

But I guess if you know that it is not really a big deal because Average Accurcy = Micro Precision = Micro Recall = Micro F1

True, that avg accuracy would give micro precision or recall.

However, it could be a nice enhancement either to give additional option as average="macro" or average="micro". And also document that micro precision/recall/F1 can be computed just using Accuracy.

[failable]

What should I do if I get this error in a binary classification and the model output does not predict any positives?