Skip to content

add Facebook MMS TTS with expanded language support - #82

Merged
andimarafioti merged 7 commits into
huggingface:facebook-mms-mergefrom
rs545837:main
Oct 14, 2024
Merged

add Facebook MMS TTS with expanded language support#82
andimarafioti merged 7 commits into
huggingface:facebook-mms-mergefrom
rs545837:main

Conversation

@rs545837

@rs545837 rs545837 commented Sep 4, 2024

Copy link
Copy Markdown
Collaborator

This PR introduces support for multiple languages in the speech-to-speech pipeline by integrating Facebook's MMS TTS model.

Key Changes:

  • Added Facebook mms handler and arguments file
  • Added a new option to specify the language for TTS with the --tts_language flag.
  • Support for Hindi (and other languages) using the facebookmms TTS model.
  • Extended the pipeline to handle multilingual speech synthesis.

Example usage with Hindi:

python s2s_pipeline.py --recv_host 0.0.0.0 --send_host 0.0.0.0 --stt_model_name openai/whisper-medium --tts_language hi --language hi --tts facebookmms --lm_model_name meta-llama/Meta-Llama-3.1-8B-Instruct

added the support for facebook mms tts, now just the language id can be used to download the relevant facebook model.
@andimarafioti

Copy link
Copy Markdown
Member

Is this up for review or is it a draft?

@rs545837

rs545837 commented Sep 4, 2024

Copy link
Copy Markdown
Collaborator Author

Yeah, Hindi is supported with this PR, I was just curious, should we add all the other languages with the same PR as well? Or maybe just few more common ones.

@andimarafioti

Copy link
Copy Markdown
Member

I would add all the languages in TTS/facebookmms_handler.py such that we know which ones are supported by facebookmms at least

@rs545837

rs545837 commented Sep 5, 2024

Copy link
Copy Markdown
Collaborator Author

List of all the supported languages by facebook-mms is here under a toggle button, there are 1000+ languages, and adding all will also require the lm and stt selection accordingly, maybe for a start we can add all the languages supported by whisper that is likely less than 100 languages.

@eustlb

eustlb commented Sep 5, 2024

Copy link
Copy Markdown
Collaborator

We're seeing here issues entailed with having such a modular pipeline and parts that support different languages. I also think that since Whisper will be mostly used, starting by adding its supported languages makes sense here.
We could benefit from the --language flag to print warnings and fall back to default english when language set for the STT part do not match the TTS part. Nevertheless, we also have the use case where we want STT to be in another language than TTS, e.g. a translator. We could then also go back to setting directly STT and TTS language using specific flags --stt_language and --tts_language.
Maybe here starting to add a clear abstraction for the STT and TTS would make sense, as we could ensure that supported languages are provided when a new implementation is added. Writing updated docs, I am realizing that some uniformisation could clearly benefit the project. WDYT ?

@rs545837

rs545837 commented Sep 6, 2024

Copy link
Copy Markdown
Collaborator Author

Alright, I added the languages supported by both whisper and mms-tts as of now.

The list of supported languages and their respective iso codes were picked up from OpenAI Docs and MMS - Language Coverage.

It's up for review now.

@rs545837 rs545837 mentioned this pull request Sep 6, 2024
@andimarafioti
andimarafioti changed the base branch from main to facebook-mms-merge October 14, 2024 20:25
@andimarafioti

Copy link
Copy Markdown
Member

I'm merging this into an intermediate branch to fix some stuff before going to main. It's been a while since you opened the PR but there are a few things to fix

@andimarafioti andimarafioti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andimarafioti
andimarafioti merged commit f0f8a22 into huggingface:facebook-mms-merge Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants