How long did you train the model

[li-xx-5]

hello，everyone.i had spent many days to train the model.how long did you take?Is there any other way to shorten the training time

[Yeongtae]

You need 870 epoch to train waveglow model.
In waveglow paper, they train the network on 8 v100 gpus with a batch size of 24, for 580K itertation.
24*580K/16K(the number of samples on LJSpeech) = 870epoch

To reduce training time, you can reduce n_channels in wn_config from 512 to 256, expand batch size(if you use v100 to train, you can expand batch size to 10 from 3), use more gpus to run distributed.py

[li-xx-5]

thank you very much @Yeongtae

[truemanlee]

I have 2x1080ti, and the training process takes at least 5 days to get a decent result.

[yxt132]

I have 2x1080ti, and the training process takes at least 5 days to get a decent result.

did you use LJSpeech? Did you change the model size (i.e., number of nodes)?

We have 6 1080ti, the results are still noisy even after a week.

[truemanlee]

I have 2x1080ti, and the training process takes at least 5 days to get a decent result.

did you use LJSpeech? Did you change the model size (i.e., number of nodes)?

We have 6 1080ti, the results are still noisy even after a week.

Yes, I used both LJSpeech and other audios (the volume was normalized to the same level). I found that the total duration of training data (34 hours in my case) played an important role. Firstly, I normalized the mel spectrogram to range [0, 1]. Secondly, I didn't change the size of network but I used the accumulated gradient method to enlarge the batch size to 8. Finally, I don't know how noisy your results are.

[yxt132]

thanks for your response! So did you figure out what is the cutoff point of the total duration at which the model may fail to converge? Is 20 hours of data long enough?
How much improvements did you see by normalizing the mel-spectrogram?
Do you mind sharing the code of using the accumulated gradient?

[Yeongtae]

• what is the cutoff point of the total duration at which the model may fail to converge?

When you synthesize using validation text manually, you can check difference between target audio and source audio by plotting audio values.

• Is 20 hours of data long enough?

13000 sentences, 8 hours are enough in my case.
You can check the samples #71.

• How much improvements did you see by normalizing the mel-spectrogram?

It makes easy to converge.
Here is my implementation for normalization to [-4, 4].
https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/layers.py
https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/audio_processing.py
https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/inference.py
I refer to code of https://github.com/Rayhane-mamah/Tacotron-2

[truemanlee]

thanks for your response! So did you figure out what is the cutoff point of the total duration at which the model may fail to converge? Is 20 hours of data long enough?
How much improvements did you see by normalizing the mel-spectrogram?
Do you mind sharing the code of using the accumulated gradient?

Actually I didn't do much comparison experiments because of the limitations of machine. The followings are mostly based on my intuition.

I believe that the more training data is, the better results generated. In fact, the generated audios are based on the mel-spectrogram, so there are few constraints of the training data. You can just collect plenty of audios and use them to train.

As for the normalization of mel-spectrogram, it mostly served for the training process. Like some issue said, I also encountered the unstable training process with small batch size (in some case, the loss exploded and the model couldn't converge again). Whether the bigger batch size can stablize the training process remains uncertain for me. The normalization formula is
torch.clamp((20 * log10(torch.clamp(mel, min=1e-5)) + 80) / 100, min=0, max=1)
And I also clip the loss with the max value of 1000 to prevent the training loss exploding.

Sorry that I can't upload the generated audios cuz I resigned from the job and I stopped the related works. But you have 6 1080ti, so actually, you can implement the accumulated gradient method to enlarge the batch size to 24 (4 times slower compared the original paper) and remain other parameters unchanged.

Finally, the noise may be a nature property of the glow-based model. If you listen the demo audio that Nvidia released very carefully, there are still a few noise.

[yxt132]

https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/inference.py

Thank you very much! I am looking at the normalizing function in your code:

def mel_normalize(x, max_abs_value=4.0, min_level_db=-100):
return torch.clamp((2max_abs_value)(x - min_level_db)/(-min_level_db) - max_abs_value,
min=-max_abs_value, max = max_abs_value)

Based on this function, if x = min_level_db, the rescaled value is equal to -max_abs_value; and if x = 0, the rescaled value is equal to max_abs_value. Anything bigger than 0 db will be clamped to max_abs_value. It seems a little strange to me. Is this correct? Or did I understand it wrong? Thanks.

[yxt132]

thanks for your response! So did you figure out what is the cutoff point of the total duration at which the model may fail to converge? Is 20 hours of data long enough?
How much improvements did you see by normalizing the mel-spectrogram?
Do you mind sharing the code of using the accumulated gradient?

Actually I didn't do much comparison experiments because of the limitations of machine. The followings are mostly based on my intuition.

I believe that the more training data is, the better results generated. In fact, the generated audios are based on the mel-spectrogram, so there are few constraints of the training data. You can just collect plenty of audios and use them to train.

As for the normalization of mel-spectrogram, it mostly served for the training process. Like some issue said, I also encountered the unstable training process with small batch size (in some case, the loss exploded and the model couldn't converge again). Whether the bigger batch size can stablize the training process remains uncertain for me. The normalization formula is
torch.clamp((20 * log10(torch.clamp(mel, min=1e-5)) + 80) / 100, min=0, max=1)
And I also clip the loss with the max value of 1000 to prevent the training loss exploding.

Sorry that I can't upload the generated audios cuz I resigned from the job and I stopped the related works. But you have 6 1080ti, so actually, you can implement the accumulated gradient method to enlarge the batch size to 24 (4 times slower compared the original paper) and remain other parameters unchanged.

Finally, the noise may be a nature property of the glow-based model. If you listen the demo audio that Nvidia released very carefully, there are still a few noise.

thank you very much

[Yeongtae]

https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/inference.py

Thank you very much! I am looking at the normalizing function in your code:

def mel_normalize(x, max_abs_value=4.0, min_level_db=-100):
return torch.clamp((2_max_abs_value)_(x - min_level_db)/(-min_level_db) - max_abs_value,
min=-max_abs_value, max = max_abs_value)

Based on this function, if x = min_level_db, the rescaled value is equal to -max_abs_value; and if x = 0, the rescaled value is equal to max_abs_value. Anything bigger than 0 db will be clamped to max_abs_value. It seems a little strange to me. Is this correct? Or did I understand it wrong? Thanks.

Check this mel_sepctrogram codes in layers.py.

In my test, it can generate intelligible voice using griffin_lim(meslpectrogram(audio)).
In my guess, The clipped values are already large enough or small enough.

[yxt132]

https://github.com/Yeongtae/tacotron2/blob/prosody_encoder_test/inference.py

Thank you very much! I am looking at the normalizing function in your code:
def mel_normalize(x, max_abs_value=4.0, min_level_db=-100):
return torch.clamp((2_max_abs_value)_(x - min_level_db)/(-min_level_db) - max_abs_value,
min=-max_abs_value, max = max_abs_value)
Based on this function, if x = min_level_db, the rescaled value is equal to -max_abs_value; and if x = 0, the rescaled value is equal to max_abs_value. Anything bigger than 0 db will be clamped to max_abs_value. It seems a little strange to me. Is this correct? Or did I understand it wrong? Thanks.

Check this mel_sepctrogram codes in layers.py.

In my test, it can generate intelligible voice using griffin_lim(meslpectrogram(audio)).
In my guess, The clipped values are already large enough or small enough.

Thank you for your quick response, Yeongtae! Really appreciate it. How is griffin_lim compared to waveglow in terms of generated audio quality and synthesis speed?

[Yeongtae]

Lower quality, slower speed. But it can generate distinguishable voice
Just use debugging and monitoring tacotron2 results.

[yxt132]

thanks. I will try to do some debugging. but I am still puzzled about the the "mel_normalize" function in your "audio_processing.py" file. Any positive "x value" (mel db) will lead to a scaled value larger than max_abs_value and therefore clamped to max_abs_value. I am wondering if that will result in information loss.

[Yeongtae]

@yxt132 feel free. This normalization make your model easy to converge.
Human recognizes the magnitude of the audio as a log scale.
Values in clipped range preserve important information.