All losses become NaN after about 1 epoch of training

[jwookyoo]

Hi,

Thank you for sharing this great work!

When I ran the training code, I got NaN for all losses after about 1 epoch of training.
This problem is reproduced whenever I run the training code. (I have tested it three times.)

I followed the same environment setting with anaconda, and also used the same hyper-parameters.
(The only difference is that our PyTorch version is 1.7.1 and yours is 1.7.0, and all other modules are the same as yours.)

Please share your idea about this problem, if you have any. Thanks!

[anthonyhu]

Hey!

Interesting. Do you run into the same issue if you first load the weights of the encoder (from FIERY Static, the single-timeframe bird's-eye view model).

To do so, add the following lines in baseline.yml

PRETRAINED:
   LOAD_WEIGHTS: True
   PATH: './static_lift_splat_setting.ckpt'

[jwookyoo]

I loaded the weights first following your suggestion, and the training works now (without NaN)! Thanks a lot!!

[jwookyoo]

Can I ask one more question? :-) How can I train the FIERY Static weights from scratch?

[anthonyhu]

Of course. To train the FIERY Static from scratch, point the training script to the following config: https://github.com/wayveai/fiery/blob/master/fiery/configs/literature/static_lss_setting.yml

[jwookyoo]

I see. Thanks a lot!

[anthonyhu]

You're welcome!