-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
An alternative approach to the speaker encoder #484
Comments
My first reaction to the concept is that the approach with siamese networks is not good enough if it only achieves 90% accuracy. But after thinking some more, a weakness of the current speaker encoder is that we train it to treat each voice as distinct even though perceptually, we consider certain pairs of voices more similar than others. So maybe less accuracy is actually a good thing if it relates unseen voices in a meaningful way to the voices that the encoder is trained on. Also, for the encoder, a higher speaker embedding size may work better for speaker ID because it has more features to discriminate between voices. But it may perform poorly for voice cloning if the encoder relies on features that humans cannot perceive. Restrict the embedding size may force the encoder to use the more tangible features. It may also help to label highly similar voices (as perceived by human) as the same voice for training. |
Another thing to note is that the siamese network is train to say if 2 samples of voice are from the same speaker or not and to do that it uses the euclidian distance between the embedded audios and this is a simple linear layer with sigmoid that gives the score The problem with the encoder of this repo (i find) is that we can’t get this think of metrics to evaluate the model so i can’t really compare them... A second advantage is for the applications of the model itself, the encoder just embeds mels so if we want to say if 2 samples comes from same speaker, we must implement the distance threshold etc, with the siamese, the output layer says it for us and it can then have many fun applications ! |
How do you handle audio files with different sample rates? I think encoder preprocessing for this repo just resamples the source audio to the desired rate. In other words I don't see the relative advantage to the GE2E approach here? My ignorance may be showing so please explain further. |
Oh yes indeed, resampling is needed but no mel spectrogram computation i would say (this repo uses à 40-features mel-spectrogram as input so processing is resampling and melspectrogram computation) |
Oh @blue-fish another funny thing you can try with embedding vector as the synthesizer ! So... what if the embedded input is a music or something else than a real human voice ??! |
You can try this with the pretrained models of this toolbox. Either load a non-voice sample, or record something with your microphone. Then try to synthesize some text. You still get intelligible speech as output because the synthesizer is only trained to make speech. |
Good, i haven’t tested the toolbox yet in fact but good to know, i think i will test it to see which impact it can have on the voice |
So the results for my tests on siamese and the actual encoder :
Siamese encoder (64-embedding) :
In two cases, the embedding plot seems good (but slightly better for the siamese i find) Plan :
I just have a question, my pretrained Tacotron is pretrained for 22050Hz and my Waveglow vocoder too but my encoder uses 16kHz audio... do you think it can be a problem to use embedding from 16kHz to train the synthesizer on 22050Hz or no ? Another think, my encoder is 64-length embedding, is this ok as input or is it too small ? |
I agree a 16,000 Hz encoder should work with a 22,050 Hz synth/vocoder. As you say, it is just an abstract representation of the voice and there is no information being passed that depends on sample rate. From 1806.04558 (the SV2TTS paper), it looks like the output similarity may be penalized slightly but they still achieve a MOS of 3+ with an embedding size of 64. So I think 64 is sufficient unless you have a large number of speakers in your synthesizer training dataset (1,000+). I am aware that the speaker counts in the table are based on the encoder training set... but if you do not have enough speakers for the synth, I hypothesize it cannot effectively utilize the additional dimensions in the encoder output. |
Interesting approach! Some thoughts that popped into my head:
In general, i'm having a hard time to see what the siamese network should produce better embeddings? Could you elaborate on this? What am i missing? |
@mueller91 in the approach of siamese in the paper i follow, he passes raw audio as input of the model and then less processing is needed compared to this repo (where you have to process the spectrogram) The other point is that i don’t use the L2 loss but a BinaryCrossentropy loss on the final layer of the siamese because the siamese has an encoder and a « decoder » part : the decoder takes the 2 embeddings, calculate distance (i use euclidian) and after that, a 1-neuron dense layer with sigmoid gives the distance (between 0 and 1) or the probability that they are same (also between 0 and 1) and then you can just use a BCE loss on that decision Another drawback of my approach is that i can’t use unfixed length audio samples (i don’t understand why, maybe not enough training) Yes the GE2E loss with cosine similarity is a good approach too and it can be interesting to compare them with real experiments (same models, training set, metrics, ...) |
Thank you so much for your answer!
|
I am training a Tacotron-2 model with my embeddings right now (you can see the other issue « pytorch synthesizer » in this repo where i give some results / ...) But i am training a tacotron-2 which is not exactly the same as the model of this repo and training with embeddings of size 64 only so i don’t know if it’s not working (actually) because the model is (a little) different or because the embedding is too small or just because it will never work with my siamese embeddings I have no RNN but i have a GlobalMaxPooling layer so it reduces the dimension to be usable and normally the maximum of a speaker should be similar in whole audios (because it should be similar between different audios) so i don’t really understand why it doesn’t work |
To see progress on this approach and results of Tacotron-2 using this siamese encoder, you can see the #507 where i describe my training results and procedure with a tf2.0 implementation of the synthesizer (slightly different from the Tacotron of this repo) |
Please see #507 (comment) |
If you continue your experiments and achieve good performance (or just interesting inference and not only noise like for me) let me know please it can maybe help me in the tensorflow implementation ! |
Originally posted by @Ananas120 in #447 (comment)
The text was updated successfully, but these errors were encountered: