how to avoid underflow in dt in dopri5?

[simitii]

After some iterations, I see the underflow in dt exception coming from dopri5.py file. What should I look for in order to debug this problem?

Thanks in advance.

[rtqichen]

This is a problem of the ODE becoming stiff, essentially acting too erratic in a region and the step size becomes so close to zero that no progress can be made in the solver. We were able to avoid this with regularization such as weight decay and using "nice" activation functions, but YMMV. Another option is to use a fixed step size solver to get an approximate solution.

[HelenZhuE]

I got the same error unfortunately. How can I use a fixed step size solver to get an approximate solution? Can you able to provide me some code about it?

[simitii]

@HelenZhuE
just give method as parameter:
odeint(func, y0, t, method="fixed_adams")

Adaptive-step:

• dopri5 Runge-Kutta 4(5) [default].
• adams Adaptive-order implicit Adams.

Fixed-step:

• euler Euler method.
• midpoint Midpoint method.
• rk4 Fourth-order Runge-Kutta with 3/8 rule.
• explicit_adams Explicit Adams.
• fixed_adams Implicit Adams.

[jandono]

This is a problem of the ODE becoming stiff, essentially acting too erratic in a region and the step size becomes so close to zero that no progress can be made in the solver. We were able to avoid this with regularization such as weight decay and using "nice" activation functions, but YMMV. Another option is to use a fixed step size solver to get an approximate solution.

@rtqichen I am experiencing the same issues. Can you be more precise on which activation functions are considered nice and why? I have tried using ReLU and Softplus and I get the error with both.

Swapping the solver to 'rk4', solved the issue for me, however I am not sure how much performance I might lose because of this.

Additionally, it might be worth noting that even with the nans, the model was working and learning. I was only able to notice a problem and the nans once I run within the torch.autograd.detect_anomaly() scope. Can anyone explain this?

[densechen]

@simitii @jandono I got the same error, however I found that the NaN will occur while forward computing, and caused this error. And I have tried many method, including different non-linearity function, different solver, but all failed.
Can you give me same advice on how to debug this?
Best
Chen