Audio track with multiple languages spoken

[FerronN]

Hi,

I've transcribed an audio track from a video where multiple languages are spoken.
Like English, Spanish and Dutch. It recognizes the Dutch audio track which is the main language.

The funny thing is that it translates (even though transcribe mode is set) some English spoken text to Dutch and some are just ignored altogether. I read something about ASR being able to detect these kind of things? But ASR task (language=auto) are only ran when you submit them by the post endpoint?

[McCloudS]

I've seen similar behavior in bilingual shows, (a mix of english and spanish) where it will randomly translate parts of spanish to english, when as you said, the task was transcribe.

Not sure how to fix it, as that appears to be something the model is handling. Do you have any examples or links of someone addressing it?

[FerronN]

I tried implementing VAD + diarization via Pyannote Audio and the community mode. I wanted to use pyannote v4, but I couldn't make the docker image work. It had to rely on Pyton 3.11 (required by pyannote v4) and also ffmpeg 6 or newer.

But this was very hard to get together in the image. Since the nvidia image is shipped with 3.10 or 3.12 (which doesn't work for whisper).

So I put it on the low burner for now.