Skip to content

v0.38.0

Choose a tag to compare

@github-actions github-actions released this 28 Sep 14:55
· 2 commits to master since this release

Which model to use for Japanese, Korean, Chinese and Cantonese

Summo knew Chinese for all three of its multilingual speech models and
Japanese for exactly one. That blank was not neutral: Summo ranks
models on how accurate they have been measured to be on the languages
your meeting declares, so a Japanese meeting had nothing to rank on and
fell through to whatever order the catalogue happened to list. The model
page said "never measured on this language" and left you to guess.

Measured now — 100 FLEURS clips per language, character error rate,
lower is better:

whisper-tiny whisper-base sense-voice-small
Japanese 55.7 % 30.0 % 8.4 %
Korean 32.8 % 25.2 % 8.5 %
Chinese 51.8 % 44.2 % 7.4 %
Cantonese refused refused 8.4 %

Bold is new. Chinese was re-run rather than copied, and reproduced the
published figures to the decimal — which is what makes the rest of the
table worth trusting.

sense-voice-small is three to six times more accurate than the best
Whisper on every language it covers
, and it idles in less memory than
whisper-base does. For a meeting in any of those languages it is not a
preference. Those numbers are in the catalogue now, so Summo ranks on
them instead of on list order, and the model page shows them instead of
a blank.

Word error rate is deliberately absent for Japanese, Chinese and
Cantonese. Nothing separates words in those scripts, so every model
scores above 100 % and the number means nothing:
sense-voice-small returns 127 % WER at 8.4 % character error on
Japanese, which is as clear a demonstration as that gets.

Whisper cannot do Cantonese at all

Not a bad score — a refusal. Cantonese is not one of Whisper's
languages, and the Whisper entries in the catalogue claim every
language, so they are offered for a language they cannot attempt.

Nothing breaks: a decode that fails costs that line its second opinion
and the recording carries on. But a Cantonese meeting with only Whisper
installed will not produce anything usable, and until this is fixed the
catalogue will not tell you so. Install sense-voice-small for
Cantonese.

Recorded as a known limitation rather than patched over, because fixing
it properly means teaching Summo which languages each runtime refuses —
its own piece of work, and worth doing once rather than guessing at.