Repository navigation
v0.38.0
Which model to use for Japanese, Korean, Chinese and Cantonese
Summo knew Chinese for all three of its multilingual speech models and
Japanese for exactly one. That blank was not neutral: Summo ranks
models on how accurate they have been measured to be on the languages
your meeting declares, so a Japanese meeting had nothing to rank on and
fell through to whatever order the catalogue happened to list. The model
page said "never measured on this language" and left you to guess.
Measured now — 100 FLEURS clips per language, character error rate,
lower is better:
| whisper-tiny | whisper-base | sense-voice-small | |
|---|---|---|---|
| Japanese | 55.7 % | 30.0 % | 8.4 % |
| Korean | 32.8 % | 25.2 % | 8.5 % |
| Chinese | 51.8 % | 44.2 % | 7.4 % |
| Cantonese | refused | refused | 8.4 % |
Bold is new. Chinese was re-run rather than copied, and reproduced the
published figures to the decimal — which is what makes the rest of the
table worth trusting.
sense-voice-small is three to six times more accurate than the best
Whisper on every language it covers, and it idles in less memory than
whisper-base does. For a meeting in any of those languages it is not a
preference. Those numbers are in the catalogue now, so Summo ranks on
them instead of on list order, and the model page shows them instead of
a blank.
Word error rate is deliberately absent for Japanese, Chinese and
Cantonese. Nothing separates words in those scripts, so every model
scores above 100 % and the number means nothing:
sense-voice-small returns 127 % WER at 8.4 % character error on
Japanese, which is as clear a demonstration as that gets.
Whisper cannot do Cantonese at all
Not a bad score — a refusal. Cantonese is not one of Whisper's
languages, and the Whisper entries in the catalogue claim every
language, so they are offered for a language they cannot attempt.
Nothing breaks: a decode that fails costs that line its second opinion
and the recording carries on. But a Cantonese meeting with only Whisper
installed will not produce anything usable, and until this is fixed the
catalogue will not tell you so. Install sense-voice-small for
Cantonese.
Recorded as a known limitation rather than patched over, because fixing
it properly means teaching Summo which languages each runtime refuses —
its own piece of work, and worth doing once rather than guessing at.