Document torch.acos() behavior near -1 and 1

[syinari0123]

I created a network using torch.acos() and encountered a Nan error during its training.

The reason is that an input value to torch.acos () approaches 1 or -1, causing the grad to diverge, and resulting in value.grad = Nan.

I know that torch.acos () can only handle values ​​in the range from -1 to 1, and limited the input value's range from -1 to 1 using clamp(min=-1,max=1). However this error has occurred.
Finally I solved this by limiting input from -1 + eps to 1-eps with eps = 1e-7.

I think that many users would encounter this error.
I suggest "to alert this in the explanation of torch.acos() function in document" or "to change function to explicitly limit the range of values ​​by including eps as argument".

[Solacex]

Thanks a lot !

[KeremTurgutlu]

Thank you !

[maxjiang93]

Thanks! Would have been nice if they adopted your feature request. Desperately finding NaNs in networks half-way through the training is not fun... I learned this the hard way... The bug boils down to acos() function of 1.

[colesbury]

You should use https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly to help find where NaNs occur in your computation.

We're not going to change the behavior of torch.acos but PRs improving the documentation are very welcome.

[HapeMask]

Since this is currently the top google result for "pytorch acos nan grad" I figured I'd share a temporary workaround / hack to unblock people:

def acos_safe(x, eps=1e-4):
    slope = np.arccos(1-eps) / eps
    # TODO: stop doing this allocation once sparse gradients with NaNs (like in
    # th.where) are handled differently.
    buf = th.empty_like(x)
    good = abs(x) <= 1-eps
    bad = ~good
    sign = th.sign(x[bad])
    buf[good] = th.acos(x[good])
    buf[bad] = th.acos(sign * (1 - eps)) - slope*sign*(abs(x[bad]) - 1 + eps)
    return buf

If #16329 is resolved, it would become much shorter:

def acos_safe(x, eps=1e-4):
    sign = th.sign(x)
    slope = np.arccos(1-eps) / eps
    return th.where(abs(x) <= 1-eps,
                    th.acos(x),
                    th.acos(sign * (1 - eps)) - slope*sign*(abs(x) - 1 + eps))

[RSKothari]

@HapeMask I ran a test case on your code snippet. I'm not sure if this is the behaviour you're expecting from acos.

import torch as th
import numpy as np
A = th.ones(5,5) + 0.1
acos_safe(A)

The output:

[HapeMask]

@HapeMask I ran a test case on your code snippet. I'm not sure if this is the behaviour you're expecting from acos.

import torch as th
import numpy as np
A = th.ones(5,5) + 0.1
acos_safe(A)

The output:

That’s expected, the function is only defined on [-1, 1] and this is simply a linear extrapolation outside the bounds.

[atnikos]

Here is another quick workout which got me through this:

epsilon=1e-8 
torch.acos(torch.clamp(array, -1 + epsilon, 1 - epsilon))

@athn-nik thanks for the tip! I was using float32 data types, so 1e-8 was too small. The minimum value of epsilon that worked for me was 3e-8, anything lower than that and torch.acos(1-epsilon) evaluates to 0.

epsilon=1e-7 worked out to be a good value.

[jungerm2]

Any progress on this? The epsilon hack works well enough, I'm just surprised this issue hasn't gained more attention...