You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I tried to train the first stage using the LJSpeech dataset provided by developer, with the Config file set as default. However, mel loss decreases to 0.5 and becomes NaN after 25 epochs. How does this happen?
The text was updated successfully, but these errors were encountered:
I am facing a similar issue. I am trying to reproduce the results of the paper and training on LJSpeech with a single GPU. As soon as the training starts the TMA stage, within 1-2 epochs the Gen and Dis loss start blowing up and eventually they NaN. I am using a batchsize of 16 and a learning rate of 1e-4. This is in the first stage of training.
Can you let me know how to stabilize this part of the training?
Perhaps issue #254 as well as its connected PR #253 could solve this - it did solve NaN value errors for me, although it was for 2nd stage training on a single GPU.
I tried to train the first stage using the LJSpeech dataset provided by developer, with the Config file set as default. However, mel loss decreases to 0.5 and becomes NaN after 25 epochs. How does this happen?
![1716028326113](https://private-user-images.githubusercontent.com/125865263/331792861-68ff1dfe-21d1-419c-9279-9e1f3b7685c2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU1MTY5NzIsIm5iZiI6MTcyNTUxNjY3MiwicGF0aCI6Ii8xMjU4NjUyNjMvMzMxNzkyODYxLTY4ZmYxZGZlLTIxZDEtNDE5Yy05Mjc5LTllMWYzYjc2ODVjMi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTA1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkwNVQwNjExMTJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT03MzgzY2EyNmYzNmIxZTEyNWNiOGRhZWNlZTdiMTFiM2NmMzI0ZmQ3ZDg1MWE0MjFlYzk0OGE2NjIwYmFjNzg4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.wKU9oF5Ys_U9yUpJv5ipJ5C21XT_5qiXi2UWorgncD0)
The text was updated successfully, but these errors were encountered: