OmniVoiceCrispAsr - Language assignment does not work

[subof]

At the moment, we have a menu where you can select the target language. But nothing changes when you select it, and you can hear the strong accent of the original and so far it's just a decoration. You need to configure this feature.

[niksedk]

CrispASR is not ready for this yet, some patience :)
Hopefilly in a week or so...

[subof]

CrispASR is not ready for this yet, some patience :) Hopefilly in a week or so...

Okay, but I would ask you not to close this topic while we wait for a decision. Thank you.

[CrispStrobe]

The CrispASR side is fixed on main now (will be in the next release). It was dead on three surfaces at once, so thanks for keeping this open — but there's one piece left that lives in SE, and one part of the report I could not fix.

What was broken on our side

1. The CLI adapter applied the language only at startup, never per request. Since SE runs one persistent server for the whole session, the menu could never change anything after the first line — even though the server has parsed a language field on /v1/audio/speech since Improve ToolBox PtsToTimeString #249/fix format specifier #304.
2. The session C-ABI's omnivoice arm never applied it at all (so the Python/Go/Flutter bindings had no language knob by any route).
3. The runtime dropped whatever string arrived straight into the model's <|lang_start|>…<|lang_end|> prompt slot without resolving it. OmniVoice's own _resolve_language() is valid ISO 639-3 id → lowercase English name → None; we did none of it, so de-DE or a typo conditioned the model on tokens it never saw there and silently looked like it worked.

Verified on the generated codes rather than the WAVs (our output is watermarked, so byte-comparing audio measures the watermark): with an English reference clip and German target text, -l de vs none differs, -l German vs -l de is byte-identical, -l de-DE vs none is byte-identical (i.e. unrecognized really does fall back to language-agnostic, not to a poisoned prompt), and three sequential requests on one server process each honoured their own language.

What SE needs to change

OmniVoiceCrispAsr.Speak() accepts TtsLanguage? language and then never sends it — the payload is {input, response_format, speed}, and the server launch args carry no -l. Adding the field is enough:

if (language != null && !string.IsNullOrEmpty(language.Code))
{
    payload["language"] = language.Code;
}

The values in OmniVoiceLanguages are already exactly right (it's generated from the same map the model uses). Unlike voice/ref-text, language does not need a server restart — it's per request. "seed" also works now; the adapter used to drop it, so re-rendering a line couldn't be reproduced.

Unrecognized values are not fatal: the server logs e.g. language 'de-DE' is not one of the model's 646 language IDs — did you mean 'de'? and falls back to language-agnostic synthesis. One quirk worth knowing since it will look like a missing entry: the map is ISO 639-3 individual languages, so there is no ar — Arabic is arb (Standard), arz (Egyptian), ary (Moroccan) and ~20 more. That's the model's vocabulary, not a gap in the list.

The part I could not fix — the accent

@subof, your other observation ("you can hear the strong accent of the original") is a separate problem, and I don't want to claim it's solved. The language tag is now live, but I couldn't measure it reducing the reference accent: over three German sentences, whisper language-ID couldn't tell tagged from untagged output apart (one sentence moved, the other two didn't — and LID is trained to be accent-robust, so it's a poor instrument for this anyway).

The structural reason is that OmniVoice has no cross-lingual mode. A voice clone always conditions on the (reference transcript, reference audio) pair, and the transcript is positioned before the reference audio in a single token stream — so we can't drop it the way we do for CosyVoice3 (#329), where dropping it is exactly what removes the accent. Cloning an English speaker onto German text may therefore still sound English no matter what the tag says.

Two practical workarounds until someone proves otherwise: use a reference clip already spoken in the target language, or use the CosyVoice3 engine for cross-lingual dubbing, which does have a transcript-dropping path (source_lang + language).

Once the payload change lands, it'd be genuinely useful to hear whether the tag makes an audible difference to you — that's a judgement I can't make with a metric, and the knob at least does something now.

[CrispStrobe]

Follow-up, and it changes what I asked of you above: the SE-side payload change is no longer required for this to work. Still worth having, but not a blocker.

I'd said the menu can't work until SE sends "language". Rather than leave the fix waiting on that, omnivoice now guesses the language from the text it's about to speak, when and only when nobody supplied one. So today's SubtitleEdit — which sends {input, response_format, speed} and no language — gets German subtitles synthesised as German. I verified exactly that request shape against a running server: byte-identical output to an explicit -l de.

Guessing is normally the wrong instinct, so for what it's worth here's the measurement I based it on rather than a hunch. I deliberately mis-tagged German text as English (-l en): the ASR round-trip came back word-perfect and whisper still identified the audio as German at 0.984 — the same band as the correct tag (0.947) and no tag (0.998). A wrong tag costs nothing detectable, while the upside is the one OmniVoice's own docs claim ("performance is slightly better if you specify the language"). If that ever turns out to be false, CRISPASR_OMNIVOICE_AUTO_LANG=0 turns it off and it goes back to opt-in.

Guard rails, since a guess that fires at the wrong moment would be worse than none:

• An explicit language always wins. If SE does send the field, the guess never runs.
• It reads the target text only, never the reference transcript — otherwise an English reference clip would drag every German line's guess back to English, which is precisely the case that matters.
• Per request, not sticky. Line N's detection can't leak onto line N+1 on a reused server process.
• Silence when unsure. Short lines usually don't carry enough evidence, and then nothing changes from today's behaviour.

Why sending the field is still worth doing: it's exact rather than inferred, it covers the languages the detector doesn't know (it handles the major scripts and a handful of European languages, out of OmniVoice's 646), and it makes the user's menu choice authoritative instead of coincidentally-correct. The snippet in my previous comment is unchanged — just no longer urgent.

@subof — the accent caveat from my last comment still stands, unfortunately; none of this changes it. But the language selection itself should now do something for you on the next CrispASR release, without waiting on an SE change.