Is there a speaker recognition gap?

[nomonkeynodeal]

I could be missing something, but it appears there are no easy, established ways to do speaker recognition. (Speaker recognition is comparing a voice sample to an enrolled profile to match a specific individual.)

I currently do this with ERes2Net sherpa-onnx. It's very bad with short utterances and introduces latency on thin clients. Running this in audio.cpp on the server would be vastly better. This would also make it possible to run bigger models.

[0xShug0]

We already have reusable speaker encoders in the framework, including ECAPA-TDNN, TitaNet, and CAM++, but they aren’t currently exposed as standalone speaker-recognition models through the server. So there is a gap in the user-facing support. Let me think about the best way to do this.

Have you tested other models? Moving inference to the server could help thin-client latency, but improving recognition on short utterances would need separate evaluation.

[nomonkeynodeal]

Grok tells me that there are a few models I should consider:

CAM++
~7.2M
Typical EER: ~0.65–0.73%
Fastest of the good ones (RTF ~0.05 on 3 s)

ERes2Net-base (what you likely have)
~6–7M
Typical EER: ~0.84–0.96%
Medium

ERes2NetV2
~17.8M
Typical EER: ~0.61% (much better at 2–3 s)
Slower (~2–3× CAM++)

ERes2Net-large
~22M
Typical EER: ~0.52%
Slowest of this family

WeSpeaker ResNet34-LM / SimAM-ResNet
6–50M
Typical EER: ~0.4%
Medium–heavy

WavLM / w2v-BERT frontends
large
Best short-utterance research systems
Too heavy for small clients

I have not figured out how to get any of these working yet. I don't think there's any way such models can be supported without the ability to make speaker profiles. Personally I only need one profile, but it would be particularly cool one day if an assistant could automatically generate them when it encounters a new voice, and then recognize that person in the future.

[0xShug0]

WavLM and w2v-BERT are also in the framework...check out https://github.com/0xShug0/audio.cpp/tree/main/src/framework/modules/speech_encoders and https://github.com/0xShug0/audio.cpp/tree/main/src/framework/modules/speaker_encoders. The issue is ECAPA-TDNN, TitaNet, and CAM++ are mostly speaker-embedding backbones rather than generic end-user models. Pretrained checkpoints are useful for speaker verification/clustering, but exposing them alone doesn't give many user-facing tasks. WavLM/Wav2Vec2 are more generic encoders, although they usually need a task-specific head or fine-tuned checkpoint for a concrete application.