a global configuration to throw error on NaN

[soumith]

NumPy has a global setting where instead of returning NaN, one can throw a RuntimeError.
https://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html

We should have something like this in pytorch (maybe).
This task is to scope out that work, or reject the feature request.

[kdexd]

This might be helpful, following the example from their docs:

>>> np.int16(32000) * np.int16(3)
   30464
>>> old_settings = np.seterr(all='warn', over='raise')
>>> np.int16(32000) * np.int16(3)
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 FloatingPointError: overflow encountered in short_scalars

Such error raising might be helpful for pytorch when a user is playing with quantization / deep compression. This is one thing I can think of.

[samuela]

please please please

[samuela]

FWIW I've implemented a wrapped version of torch that does its best to emulate this sort of behavior: https://github.com/samuela/kindling/blob/master/kindling/nan_police.py.

In [1]: from kindling.nan_police import torch

In [2]: x = torch.ones(2) / 0 * 0

In [3]: x
Out[3]:

nan
nan
[torch.FloatTensor of size 2]

In [4]: torch.sum(x)
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-4-e7f45fec8fb4> in <module>()
----> 1 torch.sum(x)

~/Development/kindling/kindling/nan_police.py in __call__(self, *args, **kwargs)
    147         if argnan_path == []:
    148           raise Exception(
--> 149             f'Found a NaN at positional argument {i + 1} (of {len(args)}) when '
    150             f'calling `{path}`!'
    151           )

Exception: Found a NaN at positional argument 1 (of 1) when calling `torch.sum`!

I've added it to my mini-toolkit of pytorch utilities, kindling.

[ElleryL]

please add this feature

[yaysummeriscoming]

please, this would be great!

[suruoxi]

Great feature for debugging. any updates?

[soumith]

anomaly detection provides this feature: https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly

However, it's not cheap.

@samuela the simulated version you have is not cheap either right?

Shall we close this feature request with a pointer to detect_anomaly?

[samuela]

@soumith No, I don't imagine that it's particularly fast, but I also haven't benchmarked it. I use it when debugging these sorts of errors and then resume using usual torch after figuring things out.

[soumith]

in that case, I'm closing the issue.

The global configurable variable to detect anomalies including nan is:

torch.autograd.set_detect_anomaly(True)

It is going to be really slow, but there isn't really something else we are aiming to do that'll be any faster.

This can also be used as a context manager for a limited set of statements as:

with autograd.detect_anomaly():
     inp = torch.rand(10, 10, requires_grad=True)
     out = run_fn(inp)
     out.backward()

Full documentation at: https://pytorch.org/docs/stable/autograd.html#torch.autograd.detect_anomaly

[R-N]

in that case, I'm closing the issue.

The global configurable variable to detect anomalies including nan is:

torch.autograd.set_detect_anomaly(True)

It doesn't seem to raise error on nan in forward pass, or does it?

"any backward computation that generate “nan” value will raise an error"