-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Universal ParallelWaveGAN #501
Comments
There it is the config I used for anyone to replica the progress here: https://discourse.mozilla.org/t/training-a-universal-vocoder/65388/14?u=erogol
|
Thanks Eren! Is this the one for the ordinary, or the larger model? I will give it a go later this week or next week. :) |
No problem. This is the normal model. You need to change layer values in the code or add them to the config file to run a larger model. |
How come you are using a scale_stats file? Is it okay to compute one for the vocoder? Also I've never maganed to train a multi-speaker vocoder with spec_gain: 1 but 20 worked. Is it correlated with the scale_stats file? |
scale stats is the mean spectrogram frame therefore anythong about spec. computation pertains it. You can compute it yourself using ```bin/compute_statistics.py`` for your dataset. I'll share the model I've trained which will include the file for LibriTTS |
I've started a larger model training using https://github.com/erogol/TTS_experiments/tree/generic_vocoder |
I will start training the larger model too, because you mentioned that at 500K steps, the noise was still there and I guess it will be the same, especially seeing as so far with the small model architecture I have never been able to get rid of it. Is that okay? I will train on LibriTTS, do you want me to resample it to 22050 or keep 24000? Also, is it okay if I train with |
"the same" to exactly which model? Do you mean a background noise or worse than that? How large is your model? Do you train with a single sampling rate or multiple sampling-rates? It is better if you use the default values to be compatible with common models. But it is your call. |
When I trained I had background noise and then I tried with more speakers and the sound was muffled. I trained using your old ParallelWaveGAN fork.
I guess it was the default model
It was single, 22050. I took LibriTTS and downsampled it.
I will train using the default values 🙂 I will train my TTS again. I can train the smaller model if you think it is better/smarter/better use of our time. And you can train the larger one |
I've also started a training session with the current generic_vocoder branch.
And here are some of my results from the current dev branch: GDrive |
These sound nice. How long did it take to reach 675K on PWGan and what GPU? Are you training using LibriTTS? |
~3 days on a V100 but the HDD and CPU are a bit of a bottleneck on the machine I'm using. |
How many speakers does it have and how many minutes of speech for each? |
Not really sure about the number of speakers, would say around ~50. Length of speech varies from few hours to just some minutes. |
How many hours is the whole dataset and how many workers are you using for the data loading? I am using a V100 and 8 workers on a 30 hour dataset and I am lucky if I get 300k steps in 3 days after the discriminator kicks in, with a batch size of 10 and a length 27500. |
~40 hours of data. I'm using 4 workes each with OMP_NUM_THREADS=1 as prefix when training. Using the default config values, batch_size: 6 and sq_len: 25600. But for now I'm giving up on training a pwgan vocoder. Can't get rid of the cracking and hiss sounds. The MB MelGan vocoder yielded better results for me even though the sound produced sounds a bit more tinny/hollow. |
I am training a larger PWGAN model. After discriminator, it raised an error when I was PTO and now I restarted it. However, from the first sight, I can tell the results look better even without discriminator training. @sanjaesc can you share some exampled with MBMelGAN? BTW in the same paper, they use a larger MelGAN model with better results. So maybe you can try that one. They only increase the model's receptive field compared to the original MelGAN. |
I am very sorry for not having time to train universal (and for the off-topic to follow). I have been testing PWGAN with single speaker, but I have had trouble with getting it to converge. Before the discriminator, it produces waveforms that sound acceptable, but when the discriminator kicks in, it sounds more metallic, especially when the speaker glottalizes. I've tried different batch sizes but it did not really help. I also tried to start training the discriminator at 100K steps instead but I got no noticeable improvements, either. Is it worth to pretrain the generator for 400K steps instead? I wonder why it does not want to converge, the PWGAN on LJSpeech sounds very good on 675K steps and my dataset has considerably better audio than LJSpeech. I did not have a lot of time to try the generic_vocoder branch. This is the config I use and I will use the same when I train the LibriTTS PWGAN, unless something is wrong with it. I decided I will go for 1100 and 275 instead, because in my tests, 1024 and 256 give me pronunciation issues with the TTS (all other settings are the same). This config here does not have normalization stats, though (because I have not used them with my TTS). Do they help a lot? I did not want to compute them, because I wanted more variety in the speech. I will switch to universal training this week. 🙂
|
@erogol are you training LibriTTS using the config in the generic_vocoder branch, or the one above and the extra layers? What batch size do you use? I think I will switch to universal now too. |
I am training the larger model now too. I am training for 275 and 1100 and for 22050 only, in hopes for saving some time, but it is taking so long. I could only fit a batch size of 8. This is the config I use.
|
@erogol can you give some info on the setup you're training on? Number of GPUs and batch size? |
batch 6 gpu 3x1080ti but the model did not work well even though it converged better now i started a new training with a constant sampling rate and a 6 block resnet on provided spectrograms before the vocoder as in the wavernn we use. |
Thanks a lot, it helps to know. Mine was also not good. But I also trained on one GPU and it was extremely slow. |
Maybe we need to add WaveRNN into Mozilla TTS the list of vocoders until we have something comparable. Is there anyone willing to do that? |
@erogol Do you think increasing the resolutions in STFT loss may help? I think (I am not sure) I read a paper where they increase them. |
What's the error when you do: waveform = vocoder_model.inference(torch.FloatTensor(ap_vocoder._normalize(mel_postnet_spec.T).T).unsqueeze(0)) |
If I do |
you can see the model release under wiki/released models. I close this issue. Thanks for your discussion and help. Feel free to comment here if you encounter with a problem. |
Thank you! Can't believe we finally have a universal GAN! For everyone else, to run it I had to take the fullband melgan model from TTS_experiments/vocoder/models and the definition in generic_utils.py (thanks Eren). I wasn't able to get the interpolation to work though, I am probably doing it wrong. @WeberJulian did you get it to work with 16khz or 22khz? |
Great thank you @erogol ! |
@george-roussos you should check the latest glow-TTS example which uses interpolation for sampling rate. https://colab.research.google.com/drive/1NC4eQJFvVEqD8L4Rd8CVK25_Z-ypaBHD?usp=sharing |
So I got the generic vocoder model to work with the 21k sr TTS model from this notebook thanks to interpolation. But I observed somthing weird, the voice sounds better if I don't denormalze with the TTS's ap and then normalize with the vocoder's ap. Am I doing something wrong @erogol ? Or is it just a matter of subjective preference ? mel_postnet_spec = mel_postnet_spec.T
#mel_postnet_spec = ap._denormalize(mel_postnet_spec)
mel_postnet_spec = torch.tensor(mel_postnet_spec)
mel_postnet_spec = interpolate(mel_postnet_spec, (1,21050/24000)).numpy()
#mel_postnet_spec = ap_voco._normalize(mel_postnet_spec) |
And by the way @george-roussos I got the same error as you |
you need to change the display sample-rate to have a more fair comparison. That is why your samples sound deeper. In the Glow-TTS notebook, it sounds better for me with denorm |
You mean the rate parameter of the IPython.display.display function ? |
Thanks guys! I was able to get interpolation to work on a TTS trained with mean-var, but no luck with TTS without mean-var :( Thanks for the vocoder Eren, appreciate it a great deal. |
Hey @erogol did you train the fullband melgan on 1 GPU with a batch size of 48? I am trying to train a run without mean-var on a V100 and it is soooooooo slow... Within 9 hours it's done, like, 25K steps. If I reduce the batch size to 16 it goes back to usual GAN training speeds, but I am afraid it might not be enough for learning. What would you suggest? |
@george-roussos have you checked your CPU usage ? If you're on the P3 aws instance, you could be CPU bottlenecked. |
I use a google instance. Have you tried it with 48? Is it supposed to be as fast as 16? |
Ok, no I haven't try this specifically, but I remeber beeing heavily CPU bound on a deep learning task because the v100 instance had only 8 vCPUs. So it's worth checking both CPU and GPU usage (htop and nvidia-smi respectively) |
That is how many I have! I have 8. Does it need more? Oh man. 😩 |
No not necessarily, it depends on the task. Just check your CPU/GPU usage to know ^^ |
Thanks. I tried with 16 vCPUs instead but it was the exact same time. 2K steps in one hour with a batch size of 48. |
yes 1GPU and 48 batch size. I agree that you might be bottlenecked by CPU. You can enable the memory feature cache ( |
Thanks, well that sucks. How many CPUs do you have? I tried with 16 but it was the exact same time (1 hour for 2000 steps). use_cache is enabled yeah |
what are step_time and loader_time in your console logs? |
It varies. It can either be something like |
If cpu usage is 100% on all cores, you're CPU bottolnecked. Your GPU usage is probably low (you can check by typing watch -n 0.1 nvidia-smi). Increasing the number of workers won't help sorry |
I thought the same (yes GPU usage is low). But when I tried 16 cores instead, it was still the same times (16 CPUs also showed 100% usage) |
I ran into the same issue with the CPU being the bottleneck. Using the prefix |
Yes! This seems to do things (now it's not 100% usage CPUs). I tried it so many times but I guess I wasn't putting it in the right order. Thanks. 🥇 |
Hi, as Eren requested, this is an issue to follow progress of the training a larger PWGAN model for multiple speakers.
The text was updated successfully, but these errors were encountered: