Feature request: Support multilingual Nemotron-3.5 streaming ASR (prompt_index input)

[paoloantinori]

Summary

Add support for the multilingual variant of Nemotron streaming ASR (nvidia/nemotron-3.5-asr-streaming-0.6b), which supports 40 language-locales with auto-detection. The English-only Nemotron is already supported (PR #3044). The multilingual variant adds a single extra encoder input tensor: prompt_index.

Background

The multilingual Nemotron model is the same architecture as the English one, with one difference: the encoder accepts a 6th input prompt_index (int64, shape [batch]) for language conditioning. When prompt_index=101, the model auto-detects the language. Specific language codes (e.g., Italian) can also be passed for better accuracy.

The ONNX files are already pre-exported and available:

• Model: pantinor/nemotron-3.5-asr-streaming-0.6b-onnx on HuggingFace
• Source: altunenes/parakeet-rs (Rust project that already supports this model)

What needs to change

The changes are minimal — auto-detect the multilingual variant from the encoder ONNX and pass prompt_index:

1. sherpa-onnx/csrc/online-transducer-nemo-model.cc

• Add bool is_multilingual_ flag, auto-detected by checking if encoder has a 6th input named prompt_index
• Add int64_t prompt_index_ member (default: 101 = auto)
• In RunEncoder(), when is_multilingual_ is true, add 6th tensor to the inputs array:

  if (is_multilingual_) {
    Ort::Value prompt_index_tensor = Ort::Value::CreateTensor<int64_t>(
        allocator_, &prompt_index_, 1);
    // inputs becomes 6 elements instead of 5
  }

2. sherpa-onnx/csrc/online-transducer-nemo-model.h

• Add SetPromptIndex(int64_t index) method
• Expose is_multilingual() getter

3. Java API (OnlineTransducerModelConfig.java or OnlineModelConfig.java)

• Add optional language field (string like "it-IT", "auto")
• Map language codes to prompt indices using the model's prompt_dictionary

Key differences from English Nemotron

Property	English	Multilingual
Encoder inputs	5	6 (extra prompt_index)
cache_last_channel dim2	70	56
Vocab size	~1024	13088
Tokenizer	tokens.txt	tokenizer.model (SentencePiece)
Languages	English only	40 language-locales

Reference implementation

The parakeet-rs Rust project already supports this model: https://github.com/altunenes/parakeet-rs

Their implementation wraps the encoder with a EncoderWithPromptWrapper that:

1. Detects multilingual from the prompt_index input
2. Creates a one-hot language vector (128-dim)
3. Concatenates it to encoder output and runs through prompt_kernel MLP

In sherpa-onnx, the prompt_index fusion is already baked into the exported ONNX encoder, so the runtime only needs to pass the tensor — no MLP logic needed.

Verified ONNX encoder inputs

Inputs:
  processed_signal:           [batch, 128, time]
  processed_signal_length:    [batch]
  cache_last_channel:         [24, 1, 56, 1024]
  cache_last_time:            [24, 1, 1024, 8]
  cache_last_channel_len:     [1]
  prompt_index:               [batch]           ← THIS IS THE ONLY ADDITION
Outputs:
  encoded:                    [batch, 1024, time]
  encoded_len:                [batch]
  cache_last_channel_next:    [24, 1, S, 1024]
  cache_last_time_next:       [24, 1, 1024, C]
  cache_last_channel_len_next: [1]

Use case

We are integrating multilingual streaming ASR into an Android on-device transcription app (Anti-Vocale). The Italian language support is critical for our users. We currently use the offline Parakeet TDT model via sherpa-onnx and would love to add real-time streaming with the multilingual Nemotron model.

We are happy to contribute a PR if this feature request is accepted.

Thank you for the excellent library! 🙏

[csukuangfj]

We are happy to contribute a PR if this feature request is accepted.

Yes, community contribution is highly appreciated.

---

The following lists some suggestions for the design.

1. Keep the user-facing API unchanged.

Add int64_t prompt_index_ member (default: 101 = auto)

Please don't do that.

The prompt index value should be passed from the recognizer to the model.
Also, the numerical value of the prompt should be an implementation detail. Users provide strings such as en, fr, etc., to specify the language. An empty string means auto.

Please use

sherpa-onnx/sherpa-onnx/csrc/online-stream.h

Line 123 in 6a9965e

void SetOption(const std::string &key, const std::string &value);

and

sherpa-onnx/sherpa-onnx/csrc/online-stream.h

Line 124 in 6a9965e

bool HasOption(const std::string &key) const;

sherpa-onnx/sherpa-onnx/csrc/online-stream.h

Line 128 in 6a9965e

const std::string &GetOption(const std::string &key) const;

Users set the prompt for a given stream. Different streams can use different prompts.

Add SetPromptIndex(int64_t index) method

Please don't add this method.

Add optional language field (string like "it-IT", "auto")

Please don't add this field.

tokenizer.model (SentencePiece)

Please convert it to a tokens.txt.

[JulianPscheid]

We'd like to take this on, unless @paoloantinori is already working on it. Happy to coordinate or just review instead if you are.

Context: we ship a meeting app that runs Nemotron streaming on-device via CoreML on Apple platforms, and we're bringing Windows and Android up on sherpa-onnx through the Flutter package. The multilingual variant is the piece we need, so we'd be testing this against long real-world sessions.

@csukuangfj your design works for us. Language as a string on the stream, index stays internal, tokenizer.model converted to tokens.txt in the export script.

One question before we start: where should the language-to-index mapping live? Baked into the model metadata at export, or a separate file next to tokens.txt?

If that sounds right we can have a PR up within a week.

[paoloantinori]

Hi there. I haven't started this yet so if you guys are ready to go, I'd say go for it!

[JulianPscheid]

PR is up: #3671

Follows the design discussed here: language as a per-stream string through the existing stream option API, prompt index internal, multilingual detected from the encoder graph, tokens.txt conversion in the export script.

@csukuangfj one open point from above: the language-to-index mapping currently ships in the encoder ONNX metadata. If you'd rather have it as a separate file next to tokens.txt, it's a small change, happy to adjust in review.

@paoloantinori thanks again for letting us run with it. If you have a setup to test against, the PR description has the validation details, more eyes welcome.

[csukuangfj]

@JulianPscheid

Thank you for your contribution! We are reviewing it.

the language-to-index mapping currently ships in the encoder ONNX metadata.

That is fine. It uses the same approach as how we handle the language map for multi-lingual whisper models in sherpa-onnx.

[csukuangfj]

Please see also #3732