feat(asr): FunASR (SenseVoice) as an opt-in alternative ASR backend (#182) - #191
Conversation
…182) FunASR is an all-in-one multilingual ASR (50+ languages, punctuation, optional cam++ speaker diarization). ASR is already pluggable, so this is a new ASRBackend: - FunASRBackend (id 'funasr'): deferred funasr import in is_available() (reports an install hint when absent — opt-in, NOT a hard dep); _ensure_model loads AutoModel(SenseVoiceSmall + fsmn-vad); transcribe() normalises output. - _normalize_funasr(): pure, defensive normaliser → OmniVoice's {chunks, segments, language} shape (handles VAD sentence_info with ms timestamps + optional speaker, single-utterance fallback, strips SenseVoice rich tokens). Unit-tested without funasr installed. - Registered in _REGISTRY → auto-appears in /system/asr-backends → the Settings ASR picker, with availability/install hint. WhisperX stays the default. Phase 2 (future): wire FunASR's cam++ speaker ids into dub diarization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis PR adds ChangesFunASR Backend Integration
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| import torch | ||
| if torch.cuda.is_available(): | ||
| torch.cuda.empty_cache() | ||
| except Exception: |
|
| Filename | Overview |
|---|---|
| backend/services/asr_backend.py | Adds FunASR/SenseVoice backend with module-level normalizer helpers; code is placed between the Registry section banner and the _REGISTRY dict rather than alongside the other backends; minor robustness gap in the single-utterance timestamp path. |
| tests/test_funasr_backend.py | Five pure-normalizer and registration tests; correctly skippable without funasr installed; doesn't cover the malformed timestamp edge case or unload(). |
Sequence Diagram
sequenceDiagram
participant Caller as Caller (segmentation)
participant GAB as get_active_asr_backend()
participant FunASR as FunASRBackend
participant AM as AutoModel (funasr)
participant Norm as _normalize_funasr()
Caller->>GAB: "bid = "funasr""
GAB->>GAB: not matched by hardcoded ids
GAB->>GAB: _REGISTRY["funasr"]()
GAB-->>Caller: FunASRBackend instance
Caller->>FunASR: transcribe(audio_path)
FunASR->>FunASR: _ensure_model()
FunASR->>AM: "AutoModel(SenseVoiceSmall, vad=fsmn-vad)"
AM-->>FunASR: self._model
FunASR->>AM: "generate(input, cache={}, language="auto", use_itn=True)"
AM-->>FunASR: raw result (list with sentence_info / text)
FunASR->>Norm: _normalize_funasr(res)
Note over Norm: Prefers sentence_info (VAD segments)<br/>Falls back to single-utterance text
Norm-->>FunASR: "{chunks, segments, language}"
FunASR-->>Caller: "{chunks, segments, language}"
Reviews (1): Last reviewed commit: "feat(asr): add FunASR (SenseVoice) as an..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # ── Registry ──────────────────────────────────────────────────────────────── | ||
|
|
||
|
|
||
| # ── FunASR (SenseVoice — all-in-one multilingual, opt-in alternative, #182) ── |
There was a problem hiding this comment.
# ── Registry ── section banner, making it visually part of the Registry section rather than grouped with the other backend classes. Every other backend lives above the banner; the FunASR block should follow the same layout so contributors scanning the file for backends see them all in one place.
| # ── Registry ──────────────────────────────────────────────────────────────── | |
| # ── FunASR (SenseVoice — all-in-one multilingual, opt-in alternative, #182) ── | |
| # ── FunASR (SenseVoice — all-in-one multilingual, opt-in alternative, #182) ── |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| ts = item.get("timestamp") or [] # [[start_ms, end_ms], ...] | ||
| start = _ms_to_s(ts[0][0]) if ts else 0.0 | ||
| end = _ms_to_s(ts[-1][1]) if ts else None |
There was a problem hiding this comment.
ts[0][0] and ts[-1][1] directly. If FunASR ever returns a timestamp list with empty inner lists (e.g. [[]]), both accesses raise IndexError which is not caught by _ms_to_s — the exception propagates out of transcribe(). Wrapping in a try/except (IndexError, TypeError) keeps the same defensive posture as the rest of the normalizer.
| ts = item.get("timestamp") or [] # [[start_ms, end_ms], ...] | |
| start = _ms_to_s(ts[0][0]) if ts else 0.0 | |
| end = _ms_to_s(ts[-1][1]) if ts else None | |
| ts = item.get("timestamp") or [] # [[start_ms, end_ms], ...] | |
| try: | |
| start = _ms_to_s(ts[0][0]) if ts else 0.0 | |
| end = _ms_to_s(ts[-1][1]) if ts else None | |
| except (IndexError, TypeError): | |
| start, end = 0.0, None |
| import gc | ||
| gc.collect() |
There was a problem hiding this comment.
import gc deferred inside unload() — gc is a zero-cost stdlib module with no install footprint, so deferring it is unnecessary. The convention throughout this file is to place stdlib imports at the top (os, re, logging). Move import gc there and drop the inline import.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/services/asr_backend.py`:
- Around line 855-872: Update the FunASR model loading to use the Hugging Face
hub and HF repo IDs: change default values for _model_name and _vad_model to the
HF mirrors "FunAudioLLM/SenseVoiceSmall" and "funasr/fsmn-vad" (respectively)
and, inside _ensure_model, call funasr.AutoModel with hub="hf" in addition to
model=self._model_name and vad_model=self._vad_model (keep disable_update=True).
This ensures AutoModel uses HF mirrors; update any env var docs if present to
reflect the new default repo IDs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d74b804c-9747-452a-b8f6-ac2b2bfc1c6f
📒 Files selected for processing (2)
backend/services/asr_backend.pytests/test_funasr_backend.py
| self._model_name = os.environ.get("ASR_MODEL_FUNASR", "iic/SenseVoiceSmall") | ||
| self._vad_model = os.environ.get("ASR_FUNASR_VAD", "fsmn-vad") | ||
| self._model = None | ||
|
|
||
| @classmethod | ||
| def is_available(cls) -> tuple[bool, str]: | ||
| try: | ||
| import funasr # noqa: F401 | ||
| return True, "ready" | ||
| except ImportError: | ||
| return False, "funasr not installed. Install with: uv pip install funasr" | ||
|
|
||
| def _ensure_model(self): | ||
| if self._model is not None: | ||
| return | ||
| from funasr import AutoModel | ||
| logger.info("FunASR loading %s (vad=%s)", self._model_name, self._vad_model) | ||
| self._model = AutoModel(model=self._model_name, vad_model=self._vad_model, disable_update=True) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Does funasr AutoModel support a hub="hf" parameter to download models from HuggingFace instead of ModelScope, and what are the HuggingFace repo ids for SenseVoiceSmall and fsmn-vad?
💡 Result:
Yes, the FunASR AutoModel explicitly supports the hub="hf" parameter, which allows you to download models from the Hugging Face Hub instead of the default ModelScope [1][2][3]. When using AutoModel, setting hub="hf" tells the loader to fetch the specified model repository from Hugging Face [4][5]. The Hugging Face repository IDs for the requested models are: - SenseVoiceSmall: FunAudioLLM/SenseVoiceSmall [6][7] - fsmn-vad: funasr/fsmn-vad [5] Example usage for loading a model from Hugging Face: from funasr import AutoModel model = AutoModel( model="FunAudioLLM/SenseVoiceSmall", hub="hf", vad_model="funasr/fsmn-vad", trust_remote_code=True, device="cuda:0")
Citations:
- 1: https://github.com/modelscope/FunASR/blob/36656aa8/docs/tutorial/README.md
- 2: https://github.com/modelscope/FunASR/blob/main/docs/tutorial/README.md
- 3: https://github.com/modelscope/FunASR/blob/main/docs/tutorial/Tables.md
- 4: https://huggingface.co/funasr/paraformer-en
- 5: https://huggingface.co/funasr/fa-zh
- 6: https://huggingface.co/FunAudioLLM/SenseVoiceSmall/raw/refs%2Fpr%2F3/README.md
- 7: https://huggingface.co/FunAudioLLM/SenseVoiceSmall
Pin FunASR downloads to HuggingFace (and switch to HF mirror repo IDs)
funasr.AutoModel supports hub="hf", which routes model downloads to Hugging Face instead of the default ModelScope behavior. If you set hub="hf" in backend/services/asr_backend.py (_ensure_model), also update the repo IDs to the HF mirrors—FunAudioLLM/SenseVoiceSmall and funasr/fsmn-vad—since the current defaults (iic/SenseVoiceSmall, fsmn-vad) are ModelScope-style and may not resolve on HF.
♻️ Pin downloads to HuggingFace
- self._model = AutoModel(model=self._model_name, vad_model=self._vad_model, disable_update=True)
+ self._model = AutoModel(
+ model=self._model_name, vad_model=self._vad_model,
+ hub="hf", disable_update=True,
+ )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/services/asr_backend.py` around lines 855 - 872, Update the FunASR
model loading to use the Hugging Face hub and HF repo IDs: change default values
for _model_name and _vad_model to the HF mirrors "FunAudioLLM/SenseVoiceSmall"
and "funasr/fsmn-vad" (respectively) and, inside _ensure_model, call
funasr.AutoModel with hub="hf" in addition to model=self._model_name and
vad_model=self._vad_model (keep disable_update=True). This ensures AutoModel
uses HF mirrors; update any env var docs if present to reflect the new default
repo IDs.
2 participants
Addresses #182 — adds FunASR (SenseVoiceSmall + FSMN-VAD: 50+ languages, punctuation, optional cam++ diarization) as an opt-in alternative to WhisperX (which stays the default).
ASR is already pluggable (
ASRBackendABC +_REGISTRY+/system/asr-backendspicker), so this is one new backend:FunASRBackend(id='funasr'):is_available()does a deferredimport funasrand reports an install hint when absent (opt-in, not a hard dep — won't bloat the default install);_ensure_modelloadsAutoModel(SenseVoiceSmall, vad=fsmn-vad);transcribe()normalises output;unload()._normalize_funasr()— pure, defensive normaliser → the{chunks, segments, language}shape the segmenter already consumes (VADsentence_infow/ ms→s timestamps + optional speaker; single-utterance fallback; strips SenseVoice<|…|>rich tokens). Unit-tested without funasr installed._REGISTRY→ auto-appears in the Settings ASR picker with availability/hint. No frontend change needed.5 tests (normaliser cases, install-hint, registry); router smoke confirms boot; CJK guard ✓.
🤖 Generated with Claude Code
Summary by CodeRabbit