Transcribe + Translate any Whisper language into English

[caio2k]

First of all, congratulations on this amazing piece of software!

My enhancement suggestion is straight-forward: I want to translate non-EU audio into English text.

While there are lots of languages covered for transcription (many thanks to Whisper), English translation is only supported for EU languages. But, as I just learned, Whisper supports transcription from supported language + translation into English at the same time. AFAIK it is only a matter of passing additional arguments to the model, and boom.
If it works, it will allow translation for many non-EU languages. And we could go the extra mile and do a TTS-STT to translate any Whisper supported text into English!

Please feel free to comment this idea!

[mkiol]

Thanks for the suggestion.

It looks that "Translate to English" option was in 3.x version but I've removed it when added Translator feature based on Bergamot engine. I thought that having two independent "translators" would be too confusing for the users and difficult to clearly represent in the user interface, but...

You're not the only one asking this, so it's clear that there is value in having this option. I will re-enable it in the next version.

BTW, Whisper can also translate from English to non-English language. It doesn't work for all languages and models but for few. When your, for instance, set "Whisper Large-v2" for Spanish and say something in English, the decoded text will be a translation to Spanish. Probably this is a "side effect" of using GTP model 🤔 I didn't figure out so far how make this "hidden" feature usable because it is not super deterministic (e.g does not work for all models).

[devSJR]

It works for German. 🤯
English (FasterWhisper Large-v3)

Edit: now, I understand what you mean. There are some sentences, that Whisper will not translate. For example, if I exchange a commonly used word for a less commonly used word with a very similar meaning. In this case, the following happens. The model is set to English. I speak German. One sentence will be in correct German. The other sentence is translated in correct English.

[mkiol]

@devSJR Exactly. It is very non deterministic. You cannot "force" engine to always translate to German. At least I didn't find any option for this. On the other hand, translation to English is very reliable and supported both by whisper.cpp and Faster Whisper.

Option to force translation to English is already implemented (2d52f56) and will be in v4.4.0.

[mkiol]

"Translate any Whisper language into English" added in: 2d52f56

Change is included in v4.4.0,