-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mixloss 出现 nan #4
Comments
我印象中之前也遇到过Nan,直观感觉是这个loss确实有点问题,和你的数据集和网络初始化有点关系,曾经试过只喂随机数,前几个step正常,然后就突然崩掉了确实也很奇怪。你换成sisnr应该就可以了正常训练,效果也是相差不大的。 |
可以先用sisnr预训练几步,初始化参数,然后在换成mixloss进行训练 |
I had this problem too. I think the non-differentiability of sqrt at 0 is the culprit here. Adding a very small value before taking sqrt, or somehow avoid passing zeroes to sqrt, can help against NaNs. |
This may be caused by the compressed coefficient \alpha, i.e., 0.3. You may as well calculate the gradient of the network output and you will get it. |
我尝试先用sisnr预训练了几步,并使用第一次保存下的模型初始化,但是mix-loss依然为nan,请问您具体是怎么做的呢 |
That works for me. I set the value to 1e-6 and I'm not sure what is appropriate to set the value. |
I've been using 1e-12. Note that the square root of 1e-6 ends up being a much larger number, namely 0.001 |
大佬,我用的是Mixloss,一运行loss就 nan.
1、LR 我已经设置很小了(0.00001);
2、没有/0 情况;请问还有可能是什么原因呢?
The text was updated successfully, but these errors were encountered: