loss explode question

[junkeon]

Hello, Thanks for your work. it is very helpful to me.
I have a question about loss exploding
I tried to train ljspeech data which you used with default setting (lr = 1e-3) and i had NaN loss issue
So i reduced lr to 5e-4 then there is no NaN loss isuue but loss exploding (normal loss : < 0.07, exploded loss : > 732M)
I know there are codes for prevent loss exploding like lr schedule, clipping however it is not working
Can you help me?

[ivanvovk]

Hello, sure. Try decrease LR scheduler gamma a bit in the first place. From 0.95 to 0.9 with default initial LR (1e-3). I used 0.95 and it was kinda borderline between normal training and training with loss explosion. And please report if it helps or not.

[junkeon]

Thank you for your advice, i will try and report it!

[ivanvovk]

Any updates, @junkeon ?

[junkeon]

As your advice, i decreased gamma varies from 0.9 to 0.5 and there are all loss explode
But it seems okay with lr = 5e-4 and gamma = 0.9 on LJ speech data
In that setting, i got l1 loss slight above 0.1 and the synthesized quality is fine

[ivanvovk]

Thank you for your feedback. And what was your batch size?

[junkeon]

sorry for late the previous feedback...
I can't remember the exact the number, it might be 48, the default setting
I can give you the number tomorrow (i am not allowed to access the server from outside), can't i?

[ivanvovk]

No-no, thank you very much, its enough information. I'm just trying to figure out why I had no loss explode problems with default setting on my server. However, I am glad that you managed to finish training, finally. Since the issue is solved, I'm closing it.