-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hallucinations and VAD [BLANK_AUDIO] Generations #45
Comments
Right now VAD isn't really working with streaming. It should be integrated into
You can try to:
This will basically make |
I can recommend Silero VAD model which has ONNX version and that's the state of the art and open source BTW https://github.com/snakers4/silero-vad/tree/master/examples/cpp |
Using the microphone record scene with VAD stop enabled and those settings you provided seem to work a lot better. Another thing I noticed is that the mic has to be pretty good for VAD to work. When I used my webcam mic the VAD struggled to stop the recording but when I used a jack mic near my face it worked relatively well within ~1 sec. after I was done talking. Thanks |
The VAD implementation is very basic. Original author of whisper.cpp recommend to use something more robust, like @BaMarcy suggested. But it is other extra dependency, which is out of this project reach. Still you can try to play with |
@atx-barnes VAD support for streaming was recently merged (see #49). In my tests this reduced hallucinations drastically. You might want to check this out. |
Tested with both small and tiny model sizes.
Using the Streaming example with VAD turned on etc. I've tried different settings and tried using a prompt to try and eliminate hallucinations and sound effects but to no avail or getting VAD to properly work I might be missing something because it treats the hallucinations of sounds like words so it struggles to turn on AD. Examples of outputs are below:
When I'm not talking and the background noise is low the following gets transcribed. Ideally, it would run inference in the background and only detect incoming audio from me talking, etc.
[BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO]
Most of the time with the tiny model, it loves to hallucinate sound effects from no audio or low background noises.
(wind blowing), (clicking), (barking)
Are there any settings that I can try that would help eliminate hallucinations from no audio or static or get VAD correctly working?
Great project, excited for any future features or updates.
The text was updated successfully, but these errors were encountered: