NAN loss after one epoch

[sjbling]

HI,
I train the model based on nuscenes with baseline.yml, but I got the nan loss after one epoch, how to train the model from scratch? Looking forward to your reply！

[markus93]

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

I also tried the same setup and have the same issue. I tried to dig a little bit deeper and found out that running means and running variances of batch normalization are getting first to -infinity and then to nan. This also causes the loss to go to nan. The loss seems to get back from the nan-values, however, the model will still output -infinities for the segmentation.

Best regards,
Markus

[anthonyhu]

Hey both of you, and sorry for the late answer. It is a known issue, and it seems like training the whole network from scratch leads to instability. The fix is to load pre-trained weight from a 1-timestep model (FIERY Static) first because training the whole future prediction model as discussed here: #8

[cs-hue]

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

After I train the model for epoch I got loss as negative value.Do you know the reason?

Best regards,
Cara

[anthonyhu]

Hey Cara, the negative loss value is due to the adaptive weighting of the losses using uncertainty. There is an additional loss term that prevents the weight from growing too large (see page 5 https://arxiv.org/pdf/1705.07115.pdf)