mdmc_average and average parameters in classification metrics (specially Dice)

[YerePhy]

Hi,

Context
Im new to torchmetrics and the documentation of these parameters is really not clear for me... I'm working with Multi-dimensional multi-class with logits or probabilities e.g. tensors with shape NCHWD. I want to compute the Dice metric 'per-class' (should give a tensor with shape NC) and the average 'per-sample' (final tensor with shape C). Im setting average='none', mdmc_average='samplewise' according with the documentation:

• average:

  • 'none' or None: Calculate the metric for each class separately, and return the metric for every class.

• mdmc_average:

  • 'samplewise': In this case, the statistics are computed separately for each sample on the N axis, and then averaged over samples. The computation for each sample is done by treating the flattened extra axes ... (see Input types) as the N dimension within the sample, and computing the metric for the sample based on that.

However, the documentation of these parameters is not clear, at least for me:

• average: Defines the reduction that is applied.
• mdmc_average: Defines how averaging is done for multi-dimensional multi-class inputs (on top of the average parameter).

Questions
It seems like the documentation is saying the same for both parameters... how these two parameters interact? Could someone be a little more specific or provide a more detailed explanation?

[MrShevan]

Hi, I think I can try to help!

I will give examples with explanations for the case of multidimension multi-class:

>>> import torch
>>> from torchmetrics import Dice
>>> 
>>> NUM_BATCHES = 4 
>>> BATCH_SIZE = 32
>>> NUM_CLASSES = 5
>>> EXTRA_DIM = 3
>>> 
>>> preds=torch.randint(high=NUM_CLASSES, size=(NUM_BATCHES, BATCH_SIZE, EXTRA_DIM))
>>> target=torch.randint(high=NUM_CLASSES, size=(NUM_BATCHES, BATCH_SIZE, EXTRA_DIM))

If you use default parameter average='micro', it means calculating metrics globally by counting the total true positives, false negatives and false positives per all classes. So in that case, you don't need to mdmc_average, cause tp, fp and fn are one-element tensors, and it's enough to count 2 * tp / 2 * tp + fp + fn (in case of Dice). Finally, you get a one-element tensor.

>>> dice = Dice(average='micro')
>>> dice(preds, target)
tensor(0.2292)

If you use default parameter average='macro', you have to set num_classes parameter, and it means calculating metrics for each label, so tp, fp and fn are tensors with dimension of number of classes. You may set mdmc_average to one of the possible options 'samplewise' or 'global', which mean averaging for multidimensional multi-class inputs like this (descriptions here). But finally you get a one-element tensor too.

>>> dice = Dice(average='macro', num_classes=NUM_CLASSES)  # Default, mdmc_average='global'
>>> dice(preds, target)
tensor(0.2300)
>>> 
>>> dice = Dice(average='macro', num_classes=NUM_CLASSES, mdmc_average='samplewise')
>>> dice(preds, target)
tensor(0.2277)

If you want to get metrics per each class, you can set average='none'. Output shape will be dimension of number of classes.

>>> dice = Dice(average='none', num_classes=NUM_CLASSES)
>>> dice(preds, target)
tensor([0.1974, 0.1875, 0.2857, 0.2519, 0.2275])

The same still works for multidimension multi-class with logits or probabilities.

[YerePhy]

Hi @MrShevan, I really appreciate your help. However, I dont know if the same holds for multidimension multi-class... I'm trying to compute Dice for tensors with the following shape:

y_pred_postprocessed.shape
torch.Size([1, 3, 96, 96]) # NCHW

y_true.shape
torch.Size([1, 3, 96, 96]) # NCHW

The clases C can be overlaped, I mean one "pixel" can be more than one class. Specifically, I'm working in tumor segmentation and the clases C are:

• C = 0: tumor core (binary mask).
• C = 1: whole tumor (binary mask).
• C = 2: enhaced tumor (binary mask).

When I do:

d = torchmetrics.Dice(average='none', num_classes=3)
d(y_pred_postprocessed, y_true)

I get:

ValueError: The implied number of classes (from shape of inputs) does not match num_classes.

Any idea? It would be nice to solve this because now I'm using another interface for computing the Dice and I wolud like to stick to torchmetrics.Metric interface for implementing all the metrics in my projects.

[MrShevan]

@YerePhy I think in your case you should modify the shape of target vector, so it has one mask with values 0, 1 and 2 (if number of classes is 3).

If your y_pred_postprocessed includes probabilities for N classes (shape is (N, C, ...)), the shape of target should be (N, ...).

Example:

>>> NUM_BATCHES = 1
>>> NUM_CLASSES = 3
>>> HEIGHT = 96
>>> WIDTH = 96
>>> 
>>> prob_preds = torch.rand(NUM_BATCHES, NUM_CLASSES, HEIGHT, WIDTH)  # NCHW
>>> target = torch.randint(high=NUM_CLASSES, size=(NUM_BATCHES, HEIGHT, WIDTH))
>>> 
>>> prob_preds.shape
torch.Size([1, 3, 96, 96])
>>> target.shape
torch.Size([1, 96, 96])
>>> 
>>> dice = Dice(average='none', num_classes=NUM_CLASSES)
>>> dice(prob_preds, target)
tensor([0.3394, 0.3464, 0.3546])

[YerePhy]

Hi @MrShevan , thanks again for your interest.

That is not possible in my case since the classes can overlap so a lot of pixels would have an not well-defined label, this is my case:

As you can see, a lot of pixels in channel 0 and 1 overlap so I must keep the target as (N, C, ...), compute per-channel metrics and average across samples...

[MrShevan]

Oh, I see… you are right.

But maybe you can count dice score separately for every channel (if number of channels is fixed) and then aggregate them? In such case, it will be possible to create mask with all classes.

[YerePhy]

I think that your suggestion is the only possible way... doing something like torch.unbind for channel dim and computing the metric for each channel... let's see what other people say.

Thanks ;)