GaussianNLLLoss doesn't support the usual reduction='none'

[almson]

The new Gaussian NLL Loss behaves differently from the other losses by not supporting the usual none mode. Instead, it still does a reduction over all dimensions except batch.

What I expect: gaussian_nll_loss(..., reduction='none') should return a tensor with the same shape as input, target, and var. The loss is essentially just 0.5 * (torch.log(var) + (input - target)**2 / var), which is what I want returned.

What happens: a scalar or a tensor of shape (N,) is returned. The implementation does .view(input.size(0), -1).sum(dim=1)

Why this matters: I use reduction='none' for custom masking, weighing, etc.

P.S. gaussian_nll_loss is missing from the nn.functional documentation.

cc @albanD @mruberry @jbschlosser

[almson]

P.P.S. The full parameter is useless and not theoretically grounded.

The main problem with a Gaussian Negative Log Likelihood that the full parameter was perhaps trying to solve is that the loss goes negative (which looks weird but doesn't actually matter). The true reason is that the likelihood (of which you're supposedly taking the negative log) is undefined. You simply can't get a probability by evaluating a gaussian at a point. You get a probability density, which is very often greater than 1. Scaling that density by sqrt(1/2pi) (which is what full does) doesn't help. The real solution is to use a PDF for the target, not a point, and integrate over it. However, we don't know this true target PDF. We can assign an arbitrary uniform distribution with a small variance, and that would translate into an arbitrary constant. Therefore, you might as well replace the full parameter with let_me_add_an_arbitrary_constant_for_you. It's better to just remove it.

[almson]

P.P.P.S. The var is clamped but the clamping doesn't mask the gradients, which seems wrong to me. It will cause divergence because var keeps being updated even if it's not being used.

[nailimixaM]

Thanks for your comments @almson, I'll look into fixing that reduction mode for you and figuring out that doc absence.

The constant added by setting the full parameter is useless with respect to optimisation (because it is a constant, and that's why it's not included by default) but it is theoretically grounded: a k-dimensional Gaussian pdf has the term sqrt[(2pi^k) x det(var)] on the denominator. Taking the log of that gives you a (constant) k*log(2pi) term + the (non-constant) variance term. Setting full just adds that constant.

The var clamping business was discussed here. I refer to @albanD @jbschlosser if you want to discuss in more detail.

[nailimixaM]

@almson After checking the clamping again, I don't think your problem will happen as a local copy of var is made to which the clamping happens:

var = var.clone()
with torch.no_grad():
. var.clamp_(min=eps)

The original var is therefore unchanged by using the loss function.

[almson]

@nailimixaM What I understand will happen is that: Assume eps is 1, var is 0.1. The gradient will be evaluated as if var=1 which, for example, will result in var.grad=1. The gradient will flow back to the real var, and after update var might be 0.09 while var.grad will continue to be 1. Eventually, var will be driven to 0. Using wrong gradients will result in divergence.

Here is the doc:

The clamping of var is ignored with respect to autograd, and so the gradients are unaffected by it.

[almson]

I found the discussion. Original implementation uses var = var + eps. I guess it would have a similar problem. The safest alternative is to simply do var.clamp(min=eps) which will zero out the gradient elements which got clamped.

[almson]

@nailimixaM

The constant added by setting the full parameter is useless with respect to optimisation (because it is a constant, and that's why it's not included by default) but it is theoretically grounded: a k-dimensional Gaussian pdf has the term sqrt[(2pi^k) x det(var)] on the denominator. Taking the log of that gives you a (constant) k*log(2pi) term + the (non-constant) variance term. Setting full just adds that constant.

I realize where that term comes from, but it's there to normalize the gaussian when integrating over it. But the integration isn't being done and would be arbitrary anyway, so including that term doesn't get you any closer to true correctness. It just leads to noise in the API and documentation.

Moreover, the term disappears if the the target PDF is assumed to be gaussian (an excellent assumption!). In that case, the loss becomes the KL loss between two gaussians, which doesn't actually have a sqrt(2pi) term.

The Gaussian KL reduces to the Gaussian (pseudo-)NLL (plus a constant) in the limit of target variance going to 0, but assuming non-negligible target variance results in an interesting K/var term. It's this term that might be interesting to include. Of note is that Gaussian KL never produces negative values (because it doesn't hack around prob densities).

I'll also add that support for full-covariance Gaussians would be a welcome (and straightforward) addition. (Formula: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Multivariate_normal_distributions)

[nailimixaM]

I found the discussion. Original implementation uses var = var + eps. I guess it would have a similar problem. The safest alternative is to simply do var.clamp(min=eps) which will zero out the gradient elements which got clamped.

I don't think the original implementation would have the problem as the addition is visible to autograd. I know that adding var = var + eps is one option used in industry, for ex. at DeepMind where a friend of mine works. However, I'd rather leave this autograd discussion to the PyTorch moderators who will know much more than I do about it - maybe open a PyTorch discussion about it?

[nailimixaM]

I realize where that term comes from, but it's there to normalize the gaussian when integrating over it. But the integration isn't being done and would be arbitrary anyway, so including that term doesn't get you any closer to true correctness. It just leads to noise in the API and documentation.

It's useful when comparing different losses, or when you want to report the actual value of the Gaussian likelihood and don't want to think about having to add klog(2pi) s everywhere.

Moreover, the term disappears if the the target PDF is assumed to be gaussian (an excellent assumption!). In that case, the loss becomes the KL loss between two gaussians, which doesn't actually have a sqrt(2pi) term.

Sure, that's a different loss function, which doesn't have such constant. That's not what this loss function is though.

Of note is that Gaussian KL never produces negative values (because it doesn't hack around prob densities).

This loss function doesn't try to avoid positive/negative values.

I'll also add that support for full-covariance Gaussians would be a welcome (and straightforward) addition. (Formula: https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Multivariate_normal_distributions)

If you mean this for the (non-KL) Gaussian NLL then I'd argue this is a very rare case as it involves matrix inversion which is usually avoided by assuming diagonal covariances. I can't comment on how common/rare it is in the KL Gaussian loss.

[albanD]

For the var question, the idea here is that this is done only for numerical stability. Of course if you clamp with a very large value, you can get very wrong values.
But in essence, what the current version does is the same (gradient-wise) as var = var + eps. The benefit is that we don't distort the var value for large enough values.

[nailimixaM]

Hi @albanD I think I've got a fix for this ready (sorry I only just got round to it) - but when I fetch from remote/master I don't seem to have any of the code for the loss function, am I missing something? Do I need to re-paste it all in?

[albanD]

Hi,

It might have been moved recently?
Looking at it it looks like this is already fixed no?

pytorch/torch/nn/functional.py

Lines 2637 to 2643 in 3fe4718

	# Apply reduction
	if reduction == 'mean':
	return loss.mean()
	elif reduction == 'sum':
	return loss.sum()
	else:
	return loss

[nailimixaM]

That's still the old buggy version (the viewing business means the shape of loss isn't the expected shape when no reduction is desired)- I'll post a PR to fix