ONNX export format support

[Darth-Carrotpie]

Thumbs up for planning ONNX support. Hope it gets prioritized more!

@Darth-Carrotpie what is your use-case of ONNX? (Just want to get some feedback)

@erogol I am trying to run models in Unity. It's environment is in C#, .NET Standard 2.1. Having a universal format model also means in the long run I can not only run models in OS agnostic manner. Of course things like tokenization and phonemization are additional hurdles, but if there are open source examples it's quite doable. For models needing tokenizers I've been using BlingFire succesfully, so I reckon there's similar phonemizer helpers / libraries for other languages beside python, including C#. Edit: things that embed python into C#, like pythonnet are convenient, though quite slow. In my case, where I have multiple models loaded and running at the same time (i.e. ~10) means that needless interpreter overhead can become a critical bottleneck. Plus it might add unforeseen debugging issues.

@Darth-Carrotpie run in unity means in the code or integrate it to Unity editor?

Also better to move this to a separate post under the Discussions

Just continuing ONNX topic which started by my ping on the roadmap issue. #378

Extending Unity editor is similar to what you refer as "code". If I understand correctly, you ask whether I'm working in Unity's company, the answer is no. I'm working on a project, where Unity is chosen as a rendering engine. However we do have some editor extensions written to aid design.
ONNX is a universal format though, it's not bound to either windows or .NET... so adding support for it would increase the reach by a lot.

[erogol]

would that work using a version of tts-server locally and calling the end-point from unity?

[Darth-Carrotpie]

Sure it would. That's not a production grade solution though.

If there's 10-20 models needed, communicating via server API complicates and bottlenecks things. In our project, we already have 4 models running natively using Microsoft.ML.OnnxRuntime and they work fine. Although they need custom code for data pre-processing and post-processing for each model, but it's minimal overhead IMO. We plan to have around 10+ models in inference mode running at the same time, which calls for a need to somewhat optimize performance in an early stage.

• So first argument is performance.
• Second argument is packaging. Having to package an API server into production is a big operations overhead which can be avoided.
• Third argument - security. Having unnecessary packages installed and a server running even if it's limited to local functionality only, allows for unintended vulnerabilities, which sooner or later stick out.

[lexkoro]

I am also interested in getting the TTS (especially the VITS model) working with onnx.
The benefit would be to get some native Windows Applications without the python dependency. For example, currently I am using pythonnet to hook into an embedded python instance to run the TTS code. Without the python dependency it should open up a fairly large target group for new apps.

[erogol]

Thanks for all the information. However, I don't think we have time for ONNX support.

We also currently don't need it internally at any level. So until then, I don't think we prioritize it.

However, if anyone is up for a PR, we are here to help.

[lumpidu]

Btw. ONNX & Torchscript would be essential for running it on Android or iOS. We are building an Icelandic open source TTS service for Android (https://github.com/grammatek/simaromur) and currently can run FastSpeech2 via torchscript. Would love to try out coqui-ai models and train an Icelandic model with it !

Have you ever run your models on a mobile phone ?

[erogol]

We haven't tried but maybe someone else from the community

[domcross]

https://github.com/MycroftAI/mimic3 uses ONNX-runtime for VITS models (not based on the Coqui-TTS implementation, though) and runs on a Raspberry Pi4 (quad-core A72 ARMv8-A) "faster than realtime". Maybe that helps?