Best practices for emotional voice cloning - any tips for preserving emotional prosody from reference audio?

[luca2125]

HI, I have a reference female voice with emotion (tense, slightly angry voice) that I need to clone to antoher language: italian however the output result is she's calm and peaceful. I ask if there is way to keep the emotion of the input reference voice.

Thank you !

[eginhard]

You didn't specify, but you're likely using the XTTS model? You can try generating multiple outputs and see if one fits better, but ultimately it's not specifically trained for emotion transfer.

[pra-dan]

@luca2125 I have found https://github.com/FunAudioLLM/CosyVoice to be the best in prosody transfer and was able to build a dubbing solution (English -> French). Luckily, it supports Italian among the 11 languages.

I am exploring Coqui-TTS because CosyVoice doesn't support Hindi. @eginhard I had the same query. I see all available models using print(TTS().list_models()) and I see a big list of models. Not all of them run successfully. Is there a family of models that I can try specifically?

[eginhard]

If there are issues with any model, please share the error message so that they can be fixed, but usually they should all work. The only Hindi model with voice cloning is XTTS.