For your information: Run VITS models from Coqui with sherpa-onnx (supporting Android, Raspberry Pi, etc)

[csukuangfj]

FYI: We have supported exporting vits models from Coqui to ONNX and run it with sherpa-onnx

sherpa-onnx supports both text-to-speech and speech-to-text and it runs on Linux/macOS/Windows/Android/iOS
and provides various APIs for different languages, e.g., C++/C/Python/C#/Kotlin/Swift/Java/Go, etc.

The following colab notebook shows how to convert vits models from Coqui to sherpa-onnx
https://colab.research.google.com/drive/1cI9VzlimS51uAw4uCR-OBeSXRPBc4KoK?usp=sharing

You can also try the exported models by visiting the following huggingface space
https://huggingface.co/spaces/k2-fsa/text-to-speech

---

We also have pre-built Android APKs for the VITS English models from Coqui.
https://k2-fsa.github.io/sherpa/onnx/tts/apk.html

[neurlang]

Does this integrate with the android Text-to-speech output api, I mean can we switch the Preferred engine to coqui?

[csukuangfj]

no, it does not.

it supports using tts models for text-to-speech locally without Internet connection.

[MXC48]

for me, the application skips words when reading: I tested the three French versions for arm64 v8 and they all skipped words.
Otherwise I find the audio quality incredible and if I had the possibility of selecting it in the system settings as the default TTS I would do so.

[MXC48]

for example, words remembered in red are not read

Please check the ./lexicon.txt and see if the word is there. If not, you can add it to lexicon.txt.

Words that are not in ./lexicon.txt are skipped.

Where is this file located? Sorry, I'm not a developer.

[csukuangfj]

For instance, for the model vits-piper-fr_FR-siwis-low, lexicon.txt can be found at
https://huggingface.co/csukuangfj/vits-piper-fr_FR-siwis-low/tree/main

[MXC48]

Can I contribute by adding words?

[csukuangfj]

Sure.

Please change the code at
https://github.com/csukuangfj/models/blob/20f97d932133bcb48646cbc8ab330cfea06f079f/.github/scripts/additional_words.py#L70

lexicon.txt is generated automagically. Please don't change lexicon.txt directly.

[csukuangfj]

It is fixed now.

No words are skipped. We use espeak-ng for models that use espeak-ng and use characters for models that use characters. lexicon.txt is no longer required for models from piper and coqui-ai/TTS.

[erogol]

@csukuangfj thanks!

[eemilk]

@csukuangfj Awesome I tried the speech recognition model example on iOS. There was also some methods already for getting vits TTS model working, are there some examples expected soon of how to use the TTS on iOS?

[csukuangfj]

Thanks for using Next-gen Kaldi!

Yes, we already support TTS on iOS. Please find the step-by-step guide for iOS in the following YouTube video.
https://www.youtube.com/watch?v=MvePdkuMNJk

Note that the above video does not reflect the latest code. We discard lexicon.txt for piper and coqui-ai/tts models.
You can find the example model at
https://github.com/k2-fsa/sherpa-onnx/blob/master/ios-swiftui/SherpaOnnxTts/SherpaOnnxTts/ViewModel.swift#L69

// https://github.com/k2-fsa/sherpa-onnx/releases/tag/tts-models
func getTtsFor_en_US_amy_low() -> SherpaOnnxOfflineTtsWrapper {
  // please see  https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/vits-piper-en_US-amy-low.tar.bz2

  // vits-vctk.onnx
  let model = getResource("en_US-amy-low", "onnx")

  // tokens.txt
  let tokens = getResource("tokens", "txt")

  // in this case, we don't need lexicon.txt
  let dataDir = resourceURL(to: "espeak-ng-data")

  let vits = sherpaOnnxOfflineTtsVitsModelConfig(model: model, lexicon: "", tokens: tokens, dataDir: dataDir)
  let modelConfig = sherpaOnnxOfflineTtsModelConfig(vits: vits)
  var config = sherpaOnnxOfflineTtsConfig(model: modelConfig)

  return SherpaOnnxOfflineTtsWrapper(config: &config)
}

func createOfflineTts() -> SherpaOnnxOfflineTtsWrapper {
  return getTtsFor_en_US_amy_low()

  // return getTtsForVCTK()

  // return getTtsForAishell3()

  // please add more models on need by following the above two examples
}