Generating non-English Tortoise models with native script support

[athu16]

EDIT: YouTube Video to demonstrate (not-so-good) voice cloning of English to Non-English speech: https://youtu.be/kzBOrMw7oBk

I cannot thank you enough for this! I just used your colab notebook to finetune the autoregressive model on this dataset, and after about two hours of training, I have probably the best-sounding offline Marathi (it's a language) TTS engine. It'll only get better with a bigger dataset and longer training. It can't replicate male voices correctly since it's a female speaker dataset, but it's certainly a great start.

So if anyone's wondering if you can use this to create non-English TTS models (with full support for your native scripts), you certainly can!

[152334H]

Interesting! I see that the dataset you've linked has only ~1.5k samples. It's encouraging to see that fine-tuning on a moderately sized dataset is enough to transfer the model to an entirely new language 👏

[athu16]

@justinjohn0306 This is a generation from the two hours of training I did yesterday on a 1.5k sample dataset: https://voca.ro/16EYt7pPYdp3
The pronounciation is a bit off, but still way better than the default autoregressor.

I did one with a 10x bigger dataset with 96 speakers, but could only train it for 1.5 hours due to colab usage limits. I'm compiling the results in a video, which I'll post here once I upload it.

[athu16]

@justinjohn0306 @152334H Here's the video demonstrating voice cloning in a non-English language. Even if you can't understand the words, you can recognize the similarity (or dissimilarity) between the cadence/timbre/tone of the voices.
There's still a lot of room for improvement, both in terms of pronunciation, as well as the actual voice replication quality. However I find it incredible that it's able to speak another language at all, considering the base model can't even do a British accent properly.

[devilismyfriend]

extremely interesting

[pheonis2]

@justinjohn0306 @152334H Here's the video demonstrating voice cloning in a non-English language. Even if you can't understand the words, you can recognize the similarity (or dissimilarity) between the cadence/timbre/tone of the voices. There's still a lot of room for improvement, both in terms of pronunciation, as well as the actual voice replication quality. However I find it incredible that it's able to speak another language at all, considering the base model can't even do a British accent properly.

did you use the default tokenizer or created a custom tokenizer for marathi language? Also, Did you also edited the tokenizer.py file to use the basic_cleaners instead of english_cleaners?

[shovonjamali]

@athu16 While training with 96 speakers, did you keep all the wavs files under the wavs folder or did you create a separate one? What was the dataset structure? If there were separate directories for each speaker, is there any change in the code?

[Snowad14]

I am training a French model for the moment on the SIWIS dataset (single very high quality speaker). On my first checkpoint the results are encouraging but it's still pretty bad. I had a total loss_gpt_total of 1.4 at the end of the 500 steps training, do you know what batch size I should use on my 3090 ?

[athu16]

I unfortunately cannot comment on the batch size. However I had the same reaction as you after training for one epoch. You'll want to train for much longer to see some good results. I trained mine for about 15 epochs (similar dataset size as yours) before it "learnt" to speak like a native person.

Also, you probably know this, but a single speaker dataset will be bad at voice cloning. If your goal is to create an offline tts engine, it's not a problem.

[Snowad14]

I finally trained my model on a dataset of 26k samples (Common Voice + SIWIS) on 1950 steps and I start to have better results, the model still has a lot of words that it can't pronounce well but the quality is really encouraging

[152334H]

glad to see more experiments with this!

I think that ultimately CLVP fine-tuning will be necessary to account for the new language (if not, samples will continue to be ranked based on an "English" interpretation of the text). Plausibly the VQVAE as well but I'd try CLVP first

...

When that's available, that is

[exxis]

EDIT: YouTube Video to demonstrate (not-so-good) voice cloning of English to Non-English speech: https://youtu.be/kzBOrMw7oBk

I cannot thank you enough for this! I just used your colab notebook to finetune the autoregressive model on this dataset, and after about two hours of training, I have probably the best-sounding offline Marathi (it's a language) TTS engine. It'll only get better with a bigger dataset and longer training. It can't replicate male voices correctly since it's a female speaker dataset, but it's certainly a great start.

So if anyone's wondering if you can use this to create non-English TTS models (with full support for your native scripts), you certainly can!

did you just create a dataset with the non-english language and let it train or did you something different/additional?

[manmay-nakhashi]

@athu16 can you share some detail on dataset prep for marathi i want to train this on hindi.

[ykli22]

Did you only fine-tune the GPT model, and does the diffusion-decoder not need fine-tuning to generate other languages?

[migueldubme]

@ykli22 could you resolve your question?

[sunny350]

@athu16 can you please suggest about tokenizer, I'm also trying to train Marathi TTS , but stuck in tokenizer file