Define output language for multilingual nvidia/parakeet-tdt-0.6b-v3 model

[rhenanbartels]

Is it possible to configure the ASR model nvidia/parakeet-tdt-0.6b-v3 to return the transcription in a specific language?

For example, if the input audio contains multiple languages, can I force the model to output the transcription in English or Portuguese only?

I am using the code provided in the repository https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3

[zhenyih]

AI generated solution, please verify

To configure the nvidia/parakeet-tdt-0.6b-v3 model to output transcriptions in a specific language like English or Portuguese when processing multilingual audio, you need to use the forced_decoder_ids parameter during generation.

Here's how to do it:

1. Load the model and processor:

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

model_id = "nvidia/parakeet-tdt-0.6b-v3"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

2. Get the language token IDs for your desired output language:

# For English
forced_decoder_ids = processor.get_decoder_prompt_ids(language="english", task="transcribe")
# For Portuguese
# forced_decoder_ids = processor.get_decoder_prompt_ids(language="portuguese", task="transcribe")

3. Use these IDs when generating transcriptions:

inputs = processor(audio, return_tensors="pt").to("cuda")
transcription = model.generate(
    **inputs,
    forced_decoder_ids=forced_decoder_ids,
    max_new_tokens=128
)
decoded_output = processor.batch_decode(transcription, skip_special_tokens=True)[0]

This forces the model to output text in your specified language regardless of what languages are present in the input audio.

[rhenanbartels]

@zhenyih I got the following erro trying to execute the code snippet below:

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

model_id = "nvidia/parakeet-tdt-0.6b-v3"
processor = AutoProcessor.from_pretrained(model_id)

ValueError: Unrecognized model in nvidia/parakeet-tdt-0.6b-v3. Should have a model_type key in its config.json, or contain one of the following strings in its name: albert, align, altclip, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, blenderbot, blenderbot-small, blip, blip-2, bloom, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, colpali, conditional_detr, convbert, convnext, convnextv2, cpmant, ctrl, cvt, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deepseek_v3, deformable_detr, deit, depth_anything, depth_pro, deta, detr, diffllama, dinat, dinov2, dinov2_with_registers, distilbert, donut-swin, dpr, dpt, efficientformer, efficientnet, electra, emu3, encodec, encoder-decoder, ernie, ernie_m, esm, falcon, falcon_mamba, fastspeech2_conformer, flaubert, flava, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, git, glm, glm4, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gptj, gptsan-japanese, granite, granitemoe, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hiera, hubert, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, jamba, jetmoe, jukebox, kosmos-2, layoutlm, layoutlmv2, layoutlmv3, led, levit, lilt, llama, llama4, llama4_text, llava, llava_next, llava_next_video, llava_onevision, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, mgp-str, mimi, mistral, mistral3, mixtral, mllama, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, owlv2, owlvit, paligemma, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_vl, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen3, qwen3_moe, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam_vision_model, seamless_m4t, seamless_m4t_v2, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip_vision_model, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, table-transformer, tapas, textnet, time_series_transformer, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xmod, yolos, yoso, zamba, zamba2, zoedepth

[saurabhav88]

+1 on this request, with production evidence and a confirmation that no workaround exists today.

We ship parakeet-tdt-0.6b-v3 on-device (Core ML, Apple Silicon) in a commercial macOS dictation app. Findings from a 10-language / ~1,700-transcription benchmark plus real user reports:

1. The forced_decoder_ids suggestion above does not apply to this model (as the OP found): it is a transformers/Whisper-style API. The TDT checkpoint has no language-conditioning input at all — the exported encoder/decoder/joint accept only acoustic and decoder-state tensors, and the model card states it auto-detects "without requiring additional prompting". NIM likewise only accepts language=multi for this model.

2. The failure this causes is real for end users. On short or ambiguous audio the model switches languages per-utterance: we have repeated end-user reports of German dictation coming out in the wrong language, including wrong-script (Greek characters) output. Post-hoc top-K rescoring can suppress wrong-SCRIPT tokens, but nothing external can reliably separate same-script languages (German vs Dutch vs English) after the joint has scored — we measured every post-hoc hard-mask approach damaging clean speech or code-switching.

3. A language token would fix it. On identical audio, a Whisper-family model with its language token forced eliminated the wrong-language/wrong-script flips entirely, at zero measured cost to clean speech or code-switching (identical WER and English-term recall, locked vs auto). Conditioning is the missing piece, not audio quality.

Request: a language-conditioned variant of the multilingual TDT checkpoint (or guidance if one is planned). The parakeet 1.1B RNNT "Prompt/Language ID" model shows the concept exists in-house, but it covers 13 languages and is a GPU/NIM artifact — not usable on the on-device TDT path where this model otherwise excels. Happy to share benchmark methodology or per-language failure examples if useful.