Language Tagging in Whisper

[Shardz4]

So, I tried providing a mixed dialect output to Whisper-base (mix of English and Hindi) and as expected it performed poorly. Does whisper really tag language based on first 30s of audio (at least what gemini gave me the answer)? Are there any models that do it in a different way?

[Yuno15-bb]

Yes, that's how the reference transcribe() works.

If you don't pass language, it detects the language once, on the first 30-second window only (pad_or_trim(mel, N_FRAMES) then model.detect_language()), and keeps that language for the whole file:

whisper/whisper/transcribe.py

Lines 143 to 156 in 8609812

	if decode_options.get("language", None) is None:
	if not model.is_multilingual:
	decode_options["language"] = "en"
	else:
	if verbose:
	print(
	"Detecting language using up to the first 30 seconds. Use `--language` to specify the language"
	)
	mel_segment = pad_or_trim(mel, N_FRAMES).to(model.device).to(dtype)
	_, probs = model.detect_language(mel_segment)
	decode_options["language"] = max(probs, key=probs.get)
	if verbose is not None:
	print(
	f"Detected language: {LANGUAGES[decode_options['language']].title()}"

So with English/Hindi code-switching, whichever language dominates the first 30 s is used for everything after.

A few things you can try:

1. Set the language yourself: model.transcribe(audio, language="hi") (or "en"), depending on which one dominates. That removes the guess on the first window.
2. Try a bigger checkpoint: base is one of the smallest models; larger ones are generally more accurate outside English.
3. Tag the language per chunk: split the audio (for example on silences), then run detection on each chunk before transcribing it, like the whisper.detect_language() example in the README.
4. Indian-language models: AI4Bharat's Vistaar project fine-tuned Whisper on 12 Indian languages (IndicWhisper): https://github.com/AI4Bharat/vistaar

[Shardz4]

Dynamic Tagging is possible right like training a frozen layer which may shift activations or output if it detects a change in language? Or am I just coping atp?

[Yuno15-bb]

Nah you're not coping, that's a real thing. People keep Whisper mostly frozen and only train a small extra part on code-switched audio. For example this ICASSP 2024 paper trains only 5.6% extra parameters on top of Whisper. It's Mandarin-English though, not Hindi.

Before training anything I'd try Oriserve/Whisper-Hindi2Hinglish-Swift. It's whisper-base fine-tuned on Hinglish, so same size as what you're using. Only catch: it writes everything in Roman script, no Devanagari.

Haven't run either myself, just went through the pages.

[Shardz4]

Thanks !!!!!!!