Unstable training

[WeberJulian]

I'm training on french mailabs with GST, speaker embedding and mixed precision enabled and my training is very chaotic. (dev)

Here is my tensorboard: (the blue part correspond to a continue_training at 10k)

The test samples speak for themselves:

Here is 13.8k (very good):
https://sndup.net/6h5y

Here is 15.7k (hell):
https://sndup.net/7qmz

here is a link to my config:
https://pastebin.com/2iCygE62

[a-froghyar]

@WeberJulian which Attention mechanism are you using? Are you using T1 or T2? For me, training with T1 GST only yielded good results with the original attention, I think Graves has a bug with T1 - #383 and the Dynamic Convolution yielded noise after 80k steps or so. Not sure about the dynamic conv but original attention definitely works, Graves just needs debugging.

[a-froghyar]

Plus, I'd actually let it train for longer, I usually only check the models after at least 20k steps

[WeberJulian]

I'm using Tacotron2 and original attention, I know it's a bit early on the training but I think the behaviour is sufficiently weird to be posted here. Have you listened to the samples ?

[a-froghyar]

Yeah you're right, it probably went wrong somewhere, does this always happen?

[WeberJulian]

No first time, but it's also the first time I enabled mixed precision.

[erogol]

you can try noam_schedule: True to let model stabilize initially with lower learning rates.

Also tb_model_param_stats:True to watch model layer stats on TensorBoard. It shows you if something is wrong with any of the layers.

[WeberJulian]

Thanks, I'll try that after trying without mixed precision

[lexkoro]

Sorry for going off-topic, but what do you think about the idea of adding also a LR value to the gradual training?

[WeberJulian]

What about ddc_r ? (we could put null when ddc is not enabled)

[erogol]

@sanjaesc Changing the LR does not make sense for me since the ultimate output length or the weights do not change.
But if you think it's useful then let's take it in a PR

@WeberJulian I remember when I did that people asked for ddc: True or False :)

[WeberJulian]

So I tried with mixed_precision: false and it works for now

[WeberJulian]

I only have one broken speaker so I can't really generalize from that... But here are the samples

Normal (male): https://sndup.net/3pwz
Normal (female): https://sndup.net/4mdf
Broken: https://sndup.net/6332

[WeberJulian]

@sanjaesc Is my broken sample representative of the issue you talked about ?

[lexkoro]

What exactly do you mean by broken? Personally I would say it sounds bit worse yeah, but without a reference audio it is hard to say.
Also is this with wavegrad? If yes, does it also sound bad with GL?

[lexkoro]

One question regarding the speaker embeddings.
Did you compute them using just the speaker-encoder? Or did you finetune the encoder first with the used dataset?

[WeberJulian]

I'll post samples of GT and GL later. Also I posted the config in the first post :) (for your speaker encoder question)