-
|
It was broken for years in the orginal coqui repo. Thanks |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
Can you test it and let me know? I haven't made any specific changes in that area, but happy to merge any fixes. |
Beta Was this translation helpful? Give feedback.
-
|
OK, I'll test it later. Thank you |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I have tested multigpu training and it does work if you dont use batch weighed sampler, and use accelerate set to true. I did not encounter any issue with ljspeech. But when I try to train on a larger dataset (libritts) I got nccl watchdog timout issues, I have tryed setting os.environ["NCCL_BLOCKING_WAIT"] = "1" but without success. How to disable timeout as precomputing the phoneme take almost 45 minutes. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for testing! You can try this to increase the timeout. Otherwise you can also do the phoneme precomputation step separately on a CPU. It only needs to be done once and the outputs are then saved and reused. Sure, PRs are always welcome! |
Beta Was this translation helpful? Give feedback.
Hi, I have tested multigpu training and it does work if you dont use batch weighed sampler, and use accelerate set to true. I did not encounter any issue with ljspeech. But when I try to train on a larger dataset (libritts) I got nccl watchdog timout issues, I have tryed setting os.environ["NCCL_BLOCKING_WAIT"] = "1" but without success. How to disable timeout as precomputing the phoneme take almost 45 minutes.
P.S The formatter for libritts dont allow to continue training if there is missing audio in libritts. I have made some modification to it if you want I can PR.
Thank you!