Add support for Voxtral

[X-Ryl669]

Describe the feature

Voxtral is a new STT (ASR) model released by MistralAI in France with open weights and an open source Apache2 license. It's much more efficient than Whisper and moreso in some languages like French where Whisper is not that good. It's based on a 3B model (there's also a 24B model available).

Would it be possible to integrate it with HA, either in Whisper integration or as another integration to replace the STT model used in local installations?

Example commands

N/A

Use cases

N/A

Anything else?

No response

[bachp]

I think it would need a change to the current architecture. As with voxtral you dont need a separate STT model. You directly feed in the audio and then get an answer or a tool call back.
I tink this would be much more efficient tough.

[Johnson145]

That's the best case I guess. I can imagine that sooner or later it may become kind of standard to directly feed audio into the model. So maybe that architecture change must get addressed sooner or later either or.

Anyway, I hope that Voxtral can already be helpful with the current approach. Simply using it as a STT service may miss the overall speed enhancement, but they also claim to have better quality results than Whisper. I'm currently using the largest Whisper model for German, but it's not really satisfying.

I've actually just started a small project trying to wrap Voxtral with the Wyoming protocol such that it can get used as a drop-in alternative to Whisper. Can't promise that I'll have enough time to actually publish a working version, but I'll give it a try within the next days.

[Johnson145]

A few days later... I've completed a working offline implementation of Voxtral to be used in HA Assist: voxtral_wyoming.

There is for sure still a lot of optimization potential, but for me it is already working quite well and the first STT results for English and German are quite promising.

If you're interested, I'd love to get your feedback of any kind.

PS: I've explicitly focused on an offline approach to allow local execution without using Cloud APIs. If you're instead looking for the latter, @mitrokun has already pointed to a good alternative.

[mitrokun]

https://github.com/fabio-garavini/ha-openai-whisper-stt-api

[Johnson145]

That's a good option, if you want to use Mistral's cloud APIs. If you are looking for an offline variant, check out my alternative proposal.