Support list of tensors in Metrics

[vfdev-5]

🚀 Feature

Related to discussion from here: Project-MONAI/MONAI#2220 (comment)

The feature request is to support list of tensors or scalars as input for metrics:
a) classification problem:

acc = Accuracy()
output = (
    [torch.rand(10), torch.rand(10), torch.rand(10)],  # predictions
    [4, 5, 1],  # ground truth
)
acc.update(output)
print(acc.compute())

b) segmentation problem:

acc = Accuracy()
output = (
    [torch.rand(10, 32, 32), torch.rand(10, 40, 30), torch.rand(10, 30, 24)],  # predictions
    [torch.randint(0, 10, size=(32, 32)), torch.randint(0, 10, size=(40, 30)),  torch.randint(0, 10, size=(30, 24))],  # ground truth
)
acc.update(output)
print(acc.compute())

c) Same for API with Engine.

cc @Nic-Ma

[Nic-Ma]

Hi @vfdev-5 ,

Thanks for raising this topic to ignite.
May I know some plan or ETA for it? We are updating MONAI metrics for it and we want to refer to your metrics as base class.

Thanks in advance.

[vfdev-5]

Hi @Nic-Ma ,

I think c) option can be rather easily done on Metric, however making it work with update/compute API is not that obvious. We are planning v0.4.5 release on the first week of june. I think we could add c) feature only there.

MONAI usage of ignite metric is with Engine only, right ?

[Nic-Ma]

Hi @vfdev-5 ,

Thanks for your update, yes, MONAI only uses metrics in engine:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/engines/workflow.py#L186
We are trying to decollate tensor batch after model forward, so the input data for metrics would be a list of dict:

[
    {"image": tensor([...]), "pred": tensor([...]), "label": tensor([...])},
    {"image": tensor([...]), "pred": tensor([...]), "label": tensor([...])},
    {"image": tensor([...]), "pred": tensor([...]), "label": tensor([...])},
    ... ...
]

The list length is batch size of model input data, the tensors in the dict are channel-first.
What do you think the best way to specify the expected data[pred, label] in ignite metrics?

Thanks.

[vfdev-5]

Hi @Nic-Ma ,

Thanks for details !

What do you think the best way to specify the expected data[pred, label] in ignite metrics?

I was thinking about supporting the following input:

metric_input = (
    [torch.rand(10, 32, 32), torch.rand(10, 40, 30), torch.rand(10, 30, 24)],  # predictions
    [torch.randint(0, 10, size=(32, 32)), torch.randint(0, 10, size=(40, 30)),  torch.randint(0, 10, size=(30, 24))],  # ground truth
)

If as a dict:

metric_input = {
    "y_pred": [torch.rand(10, 32, 32), torch.rand(10, 40, 30), torch.rand(10, 30, 24)],  # predictions
    "y": [torch.randint(0, 10, size=(32, 32)), torch.randint(0, 10, size=(40, 30)),  torch.randint(0, 10, size=(30, 24))],  # ground truth
}

Do you think it could be adapted for MONAI users to provide the data in one of those 2 formats ?

[Nic-Ma]

I see your point, your example is "dict of list", could you please also support "list of dict"? Just like the example data:

[
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    ... ...
]

Thanks.

[vfdev-5]

Basically, we would support the "tuple of lists" format, like the first example.

When we are supporting dicts, we have few conventions and mentioned in our docs that predictions should have "y_pred" key and the ground-truth has "y" key. See for example: https://pytorch.org/ignite/metrics.html#ignite.metrics.Accuracy

update must receive output of the form (y_pred, y) or {'y_pred': y_pred, 'y': y}.

Same logic is applied here :

metric_input = {
    "y_pred": [torch.rand(10, 32, 32), torch.rand(10, 40, 30), torch.rand(10, 30, 24)],  # predictions
    "y": [torch.randint(0, 10, size=(32, 32)), torch.randint(0, 10, size=(40, 30)),  torch.randint(0, 10, size=(30, 24))],  # ground truth
}

Otherwise, the keys are not supported and user should use output_transform to adapt between formats.

In your case of "list of dicts" we have to keep the same conventions like

[
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    {"image": tensor([...]), "y_pred": tensor([...]), "y": tensor([...])},
    ... ...
]

or use output_transform...

[Nic-Ma]

I see, thanks for your sharing.
@wyli We need to add some logic to convert data from "list of dict" to "tuple of list" for pred and label.
Would it be available in ignite v0.4.5?

Thanks.

[vfdev-5]

I think once we are OK with details on what to add, we can try to implement it and schedule for this release.

[Nic-Ma]

Sounds great!

Thanks.

[vfdev-5]

Hey @Nic-Ma , just to make sure, are we ok with the formats to handle described here : #2013 (comment) ?
Such that we could advance on that and probably include to v0.4.5 release.

[Nic-Ma]

Hi @vfdev-5 ,

Thanks for pinging.
I was thinking that: as you are adding the engine.state.data, maybe we can also add 2 args in ignite metrics to easily extract the y_pred and y from engine.state.data? Something like:

class Metric(metaclass=ABCMeta):
    def __init__(
        self,
        y_pred_key: Optional[str] = None,
        y_key: Optional[str] = None,
        output_transform: Callable = lambda x: x,
        device: Union[str, torch.device] = torch.device("cpu"),
    ):
        self._output_transform = output_transform
        self.y_pred_key = y_pred_key
        self.y_key = y_key

    @torch.no_grad()
    def iteration_completed(self, engine: Engine) -> None:
        if isinstance(self.y_pred_key, str) and isinstance(self.y_key, str):
            output = engine.state.data.get(self.y_pred_key), engine.state.data.get(self.y_key)
        else:
            output = self._output_transform(engine.state.output)
        if isinstance(output, Mapping):
            output = tuple(output[k] for k in self.required_output_keys)
        self.update(output)

Others look good to me so far.

Thanks.

[Nic-Ma]

Of course, I think you can enhance the metrics with your current proposal first.
Then maybe update again when engine.state.data is ready.

Thanks.

[Nic-Ma]

Hi @vfdev-5 ,

Do you have any updates here?

Thanks in advance.