hayamimi 早耳 v0.1.1 — noise robustness, measured honestly
hayamimi 早耳 v0.1.1 — noise robustness, measured honestly
v0.1.0's accuracy numbers were all measured on clean speech. This release asks what happens when the audio isn't clean — noisy rooms, background chatter, and (by user request) singing — measures it, and fixes what could be fixed.
Noise robustness
We ran the full pipeline against white, pink, and babble noise at four signal-to-noise levels for all five main languages (docs/NOISE.md). The clearest finding: background chatter is by far the most destructive noise type, and it breaks language identification before it breaks transcription — one wrong guess routes the audio to the wrong language model and the subtitles collapse.
The fix targets that exact failure: switching languages now requires two consecutive agreeing detections, so a single noise-induced misdetection no longer flips the subtitle language. In the noise evaluation this improved language identification in 15 of 40 noisy conditions with zero regression on clean audio. The trade-off is that a foreign phrase only one segment long no longer triggers a switch; lid_switch_confirm=1 restores the old instant behavior.
We also tested the obvious alternative — putting a speech denoiser (GTCRN) in front of the recognizer — and rejected it based on the same measurements: it made accuracy worse in 49 of 60 noisy conditions and even degraded clean audio. The A/B data is in docs/NOISE.md, so the decision can be re-examined if better denoisers appear.
Singing
A user asked how well hayamimi transcribes singing, so we built an evaluation set and measured it (docs/SINGING.md). A cappella Japanese lands at a kana-level character error rate of 0.24, versus 0.15 for the same singer reading the same lyrics — degraded, but readable as subtitles. English and Korean songs fall to roughly 50% error, and when language identification guesses wrong, that clip is lost entirely. If you plan to sing, pinning the language with --lang ja avoids the worst case. Accompanied music is untested and will be harder.
Reproducibility
The noise and singing evaluation sets are both built from public corpora by scripts in the repository (make_noisyset.py, make_singset.py), and the scoring scripts (eval_noise.py, eval_singing.py) are included. Unit tests grew from 15 to 23. No model changes; download_models.py output is unchanged.
日本語での要約
v0.1.0の精度はクリーンな音声での測定でした。このリリースでは、ノイズ環境と歌声でどこまで劣化するかを実測し、直せる部分を直しています。背景の話し声が最も破壊的で、文字起こしより先に言語判定が崩れると分かったため、言語切替に連続2回の一致を要求するようにしました（ノイズ40条件中15条件で判定が改善、クリーン音声への影響はゼロ）。ノイズ除去モデルを前段に挟む案は、60条件中49条件でむしろ悪化したため、測定データを公開した上で不採用としています。歌声は日本語アカペラなら朗読の約1.6倍の誤り率で字幕として読める水準、英語・韓国語の歌は誤り率5割前後でした。評価セットの生成・採点スクリプトはすべて同梱しています。
🤖 Generated with Claude Code