Add an option for Dection Models to return cost in eval mode as well (validation, test loss)

[Delaunay]

We need to get the model cost in eval mode when we compute validation loss/cost for hyper parameter search.

At the moment eval mode only returns the detection without the loss/cost.
Which mean it is used as inference mode which is different than what people might want.

We can not use training mode to compute validation because of batchnorm and dropout.

The solution with the least code modification would be to add a new argument inference_mode in forward that would be true if we only need the detection and false if we want the loss/cost.
and change if self.training to if not inference_mode.

Few classes through out the torchvision.models.vision would need to be updated like so

[elangovana]

I would be interested in this feature too.

[fmassa]

Hi,

I think adding a new argument to the constructor might be a good way of handling this.
But given that this would be a breaking change, we will be doing it after the next release of torchvision, which will happen in January.

[ryzalk]

This would be a great feature. Looking forward to it :)

[cyyever]

I need this feature

[NicolasHug]

From offline discussions with @fmassa, we need the 3 following features:

• A way for eager and scripted mode to return the same type, i.e. a tuple with losses and detections (with either losses or detections being possibly empty elements of the tuple)
• A way to return detections during training
• A way to return losses during eval

---

I think the following design might work:

• add new return_tuples param {True, False (default)}
  • possibly: add a None option that would be the default and that would warn users and tell them to explicitly set this parameter. But if we need to always support the legacy version anyway (by legacy I mean the current behaviour), this might be unnecessary.
  • When False, this corresponds to the legacy behaviour
  • When True, script and eager mode have identical behaviours (i.e. they return the same thing), as dictated by the next 2 parameters
• add new compute_losses param {'always', 'training_only' (default)}
  • if 'always':
    • return_tuples must be True
    • if targets aren't passed in eval, raise error (or at least warn)
• add new compute_detections param {'always', 'eval_only' (default)}
  • if 'always', return_tuples must be True

It's 3 new parameters, but they're all backward-compatible so users only need to set them if they need/want to. And one advantage is that it's very explicit in what it does.

We could potentially avoid introducing the compute_losses parameter and basically return the losses iff targets is not None, but I fear this might be too implicit, and since we need to introduce compute_detections anyway, it's probably cleaner to do the same for losses.

Thoughts @fmassa ?

[NicolasHug]

From a discussion with @fmassa : there is hope that torchscript will eventually support returning union types. This would allow the scripted mode to return either the losses or the detections, instead of always returning a tuple, and would make eager and scripted versions compatible with a minimum amount of backward incompatibility. It may be preferable to wait until then, as the rest of the issues will be easier to address as well.

[bfialkoff]

I think Union types are now supported see here].

Also wouldn't it be simpler to just adjust the relevant forward methods to return losses if targets is provided. Then we could avoid discussions of inference_mode and just compute the losses if the targets are provided and return the detections always.

[khalidcawl]

Any updates on this? I looked around the code (the forward function here) and couldn't find what to do if I want to get losses and detections both during training and evaluation. Even getting the losses only (both during train and evaluation) should be enough.

Any help would be appreciated

[claushofmann]

+1 for this issue. IMHO the most user-friendly solution (although I don't exactly know what would be the implications) would be to always return detections and losses if the user provides (images, targets) and to only return detections if the user provides (images), disregarding whether the current mode is training or eval.