Loss metric to use required_output_keys

[vfdev-5]

🚀 Feature

Currently, if we have custom metrics that require data other then y_pred and y, we suggest to do the following:

metrics = {
    "Accuracy": Accuracy(),
    "Loss": Loss(criterion, output_transform=lambda out_dict: (out_dict["y_pred"], out_dict["y"])),
    "CustomMetric": CustomMetric()
}

evaluator = create_supervised_evaluator(
    model, 
    metrics=metrics, 
    output_transform=lambda x, y, y_pred: {"x": x, "y": y, "y_pred": y_pred}
)

where CustomMetric is defined as

class CustomMetric(Metric):

    required_output_keys = ("y_pred", "y", "x")

The idea is to extend this for Loss metric to support required_output_keys. The main issue with Loss now is with (prediction, target, kwargs) optional input, where kwargs is a dict for extra args for criterion function.

[sdesrozis]

So, output_transform signature used in Loss initialization should be defined in a generic way in the Loss class ? and this signature is related to the criterion ?

The output dict is compared to required_output_keys and here, we need to handle extra args defined in criterion... Not so easy because requirements are not statically known.

EDIT not sure to really understand ...

[vfdev-5]

@sdesrozis yes, you are right about class params vs instance params. We can adopt two approaches:

1. if user would like to pass kwargs for the criterion, there is no way to use dict as state.output
2. in Loss we define required_output_keys=("y_pred", "y", "criterion_kwargs") and ask the user to define state.output as

state.output = {
  "y_pred": ...,
  "y": ...,
  "criterion_kwargs": {...}
}

Let's take 2nd approach.

[madhucharan]

Hello @vfdev-5 , I am new to PyTorch ignite and studied this issue for some time and understood some part of it related to loss.py but stuck with criterion. Would it be possible to explain the expectation of this issue and some information about the criterion function as well as a state? output. I am interested to work on this issue.

[vfdev-5]

@madhucharan thanks for proposing a help with this issue. Please read https://pytorch.org/ignite/metrics.html#ignite.metrics.Metric.required_output_keys about required_output_keys.
Please, see our CONTRIBUTING guidelines. You can also take a look at Quick-start and Concepts sections of our docs.

[madhucharan]

Hi @vfdev-5, Thanks for the information. I went through the code base and its functioning related to this issue.

In pytorch we generally use like this,
criterion = nn.CrossEntropyLoss()
in training
we use
loss = criterion(output, target)

Is this the criterion mentioned in the first comment? or something else. Kindly clarify this.

2. And the state.output is actually an output returned by run() function right? I would like to know why user needs to define state.output dict and how and where it is passed as an argument?

[vfdev-5]

@madhucharan well, in most of the cases a loss function in pytorch takes predictions and targets, but there are also cases where loss function can accept other params (e.g. reduction for functional cross entropy) and also inputs like sample x etc.
See the link in the first message (https://discuss.pytorch.org/t/how-access-inputs-in-custom-ignite-metric/91221).

The idea of this issue is to simplify Ignite's part of handling those functions.
Try to play with the code snippet from here : https://discuss.pytorch.org/t/how-access-inputs-in-custom-ignite-metric/91221/6

And the state.output is actually an output returned by run() function right? I would like to know why user needs to define state.output dict and how and where it is passed as an argument?

Passing a dict as the output of the evaluation step function (which is written to state.output) is a nice way to structure the output.
In most of the cases, output is a tuple of (y_pred, y_true) which are used to compute metrics.

Hope this helps to understand.

[sparkingdark]

@vfdev-5 are we going to create an extended class customMetrics where users can use custom operations like reset, update, and compute.

[vfdev-5]

@sparkingdark no, the idea is to improve existing Loss class

[sparkingdark]

Okay