More details on whisper.decode() for translation

[thejame]

I'm looking for more info on the whisper.decode() function as well as log_mel_spectrogram() so I can send flags re translation. I see how to recognize a language on input file and convert from a language but not TO a specified one. Thanks.

[misutoneko]

AFAICR the target language for translation is always english. If you need some other language you will need to do it separately.
(I've used madlad400 for english-to-finnish translation, it's ok but not without problems...)

[thejame]

transcribe() allows for language as a parameter, but I want to see what's available in decode() if it's possible.

[misutoneko]

I feel like I'm probably missing something here... but isn't it all defined in DecodingOptions class?

[thejame]

Thanks, I see the class definition but don't understand how to use the task var:

class DecodingOptions:
     # whether to perform X->X "transcribe" or X->English "translate"
    task: str = "transcribe"

I'm already using transcribe, but I don't understand how to pass it a source and target language.

[misutoneko]

In "transcribe" mode, the source and target language are the same.
In "translate" mode it's as I said, the target language can't be set, it's always english.

For example, if you have the source audio in spanish and want the resulting text to be in german, you would use "transcribe" to get the spanish transcript. You will then have to implement the spanish-to-german translation yourself, as a separate step (using opus-mt, madlad400, whatever).
(Or alternatively, if you use "translate" mode you will get the english text and you will then have to translate that to german yourself. But the timing information might suffer in this case.)
Getting the timing right for the translated version might be a bit tricky.

Also note that the language barriers are not absolute, there is some "fuzziness" about it because the training material does not always consist purely of one language.

[thejame]

Thank you, I understand now. Thanks for being patient and thorough.