Training on my data sometimes breaks on maximum number of Nans of the NLL (100)

[manifischer]

I am training DeFlow on my own generated data.
During the train I sometimes get a raise due to reaching maximum number of Nans of the NLL (100).
Also, on some of my validation images, I get Nans during training, and also on translate.py (and results are of course bad).
Do you have an idea of the root cause of those Nans, and maybe an idea of how to resolve them?
Thanks,
Mani

[volflow]

Hi Mani,

We also ran into the problem NaNs arising in some training runs.
This problem arises when some learned scaling in the network becomes too large or too close to 0, which then lead an overflow/underflow in the computation.
The easiest thing you can try to avoid this is to decrease the learning rate as we found that this stabilises the training.
An other thing you can experiment with is to bound the learned scalings, we did however not find a good way to do so without decreasing performance.

Let me know if this helps!

Best,
Valentin

[manifischer]

Thanks you,
I will try reducing the learning rate first and let you know
Best,
Mani

[manifischer]

Reducing the learning rate by a factor of 10 to 0.00001 did the job and I get no more Nans during training.
Thanks a lot
Mani

[volflow]

Great to hear!