Possible to use for real-time / streaming tasks? #2
|
Is it possible to use |
Replies: 30 comments 69 replies
|
It doesn't support real-time per se, but you could build something similar by e.g. incrementally transcribing the audio every second. |
|
how is the performance? |
|
I recently wrote a tutorial for streaming speaker-colored transcriptions combining whisper and diart (based on pyannote.audio models and supporting websockets). I'm sure many improvements can be made 😃 Here's a demo: good1.1.mp4 |
|
.js-local-storage-resumable |
|
@JennPlothow |
|
Is the character @JennPlothow generated by someone with AI? Looks like someone if faking a Brazilian girl: I saw on his/her linked in: |
|
Hello everybody I came across this api with the exact same thing in mind. I created an organization for this. iInterpret the envision is an app that can translate verbal speech in real time for phone calls or communication in person maybe with a bluetooth piece? This would be a blessing for some of my business in China to go from Mandarin to english seamlessly. If this interests you or anyone else reading this please join the organization and feel free to reach out to me via me@scottbeeker.com |
|
@jojeyh can you share your repo |
|
@jojeyh I would appreciate to see your repo. |
So how does the sentence between two silences affect accuracy? Does having shorter sentences decrease the accuracy? @joorjeh |
|
@rahulbansal16 It was a while since I used this package but I believe the npm module in question has a minimum time chunk required for accepting as valid speech segment, otherwise it considers it a spurious voice recognition and ignores. |
|
@rahulbansal16 In general, shorter audio segments in ASR will necessarily be less accurate as they have less signal (read context) to determine the right text. You might want to check out resources at https://github.com/ufal/whisper_streaming. Usually for a streaming situation you use a rolling window and update the text as you move the window. |
|
Thank you collegues. Working at the International Telecommunication Union (United Nations Specialized Agency for ICT) we would really like this for our meetings. ps. the title of the org for me is quite hard to read, 'l then i' |
|
Awesome look forward to working with you I took your suggestion and have renamed it to myLinguist. |
|
Here is another attempt for real-time streaming: rt_esl_csgo_1.mp4This is using |
|
Such a cool demo. I made a similar real-time app (+ a GUI) by splitting the audio into chunks.
Demo: https://www.loom.com/share/564b753eb4d44b55b985b8abd26b55f7?t=34 |
|
And here is another example with even more "real-time" transcription using rt_esl_csgo_2.mp4 |
|
link code please. |
It was in the main post. |
|
I've built Whisper Playground for developers to easily build real-time speech2text web apps Whisper.Playground.mp4 |
|
"Building wheel for pyaudio (pyproject.toml) ... error" |
|
@ibnbayo |
|
no module named whisper. i'm using python 3.10.7 |
|
@shekharlilly try this, it works in chrome extension, Mozilla and also from microphone |
|
I did the installation for whisper by this 'pip install git+https://github.com/openai/whisper.git ' . |
|
Mby check your python version |
|
You can check my project: https://github.com/appvoid/vosper |
|
Please see my project below, which uses the Whisper Tiny Tflite Model to implement audio streaming.. |
|
If you want reduce processing time of transcribe when you use whisper for streaming, you can use whisper decoder for get only tokens of transcribe and decode it using tokenizer. Because the buffer of audio from the streaming chunk dont have length until 30 second, and in the transcribe of whisper there temperature and logprob, and the other prob for get the best result of transcribe, it process will need more iteration, it means you will need time more longer |
|
You cant use whiper c-translate, for 1 worker, this model have RTF 0.05-0.06 for large model, this RTF will incrase if you use multi concurrent. For large model, 5 minutes audio, i get RTF 0.6 for 15 thread concurrent. GPU memory used only 3.5 gb when you set 1 thread in the parameter and CPU core can reach 15 until 50 cores. But you can limit your core until minimum 2 cores, because if you set only 1 core, the process will longer 30 seconds. |
|
This repo for whisper c-translate |
|
This is very helpful. Thanks |
|
When you use CTranslate2 and Silero, are you simply waiting for an audio gap then sending it to faster_whipser for transcription and using the model.transcribe function? Or are you using a different (maybe lower level function like generate) in CTranslate2 faster_whisper>? |
|
@Mijawel I waiting audio segment from VAD Silero. So VAD Silero will create audio segment from audio buffer, and it will transcribe using ctranslate2 model.transcribe. For the VAD, i use timestamp method from VAD Silero. VAD will create list of timestamp so incoming audio chunk will be buffered and entered into VAD, when the length of list change, you can take the first array of list timestamp for entered into transcribe model. |
|
This is my take for React.js useWhisper React hook can now do real-time transcription. Repo: Demo: (Whisper seems to can not understand my accent 😅) use-whisper-real-time-transcription.mp4 |
|
This is using the openai whisper api, right? |
|
@FerLuisxd you might want to consider using the Speech Recognition API for this. However, please note that not all browsers support it. You can find more information here: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition |
|
Hey! |
|
In essence, what we ultimately need is a Real-time Syllable Recognition engine with a mechanical keyboard precision, for example for mandarin, I get the syllables of "ni3 hao3 ren2 men2 zong1 guo2 han4 zi4" and send it to ChatGPT-4, if we have a concise Real-time Syllable Recognition engine, LLM will replace the entire speech recognition industry. ref: "Transcribe to IPA" is very important for realtime interaction application #318 (comment) |
|
Post App World requires a Syllable based Recognition Engine with a mechanical keyboard precision and with the speed which is faster than the mankind pronunciation. Daniel Povey, the developer of Kaldi 2, believes that "Proactive Mandarin Syllable Recognition Engine" can't be achieved,but I still feel it can. the ASR engines should focus more on the recognition of syllables.
|
|
Hi guys, I implemented realtime Whisper streaming for long audios in Python. Going to share it soon. |
|
I've test your project with "python3 whisper_online.py nihao.wav --language zh --model small --min-chunk-size 0.5": Typically every syllable cost the same fixed 0.3 second in mandarin, |
|
This is a truly streaming model, not a hack in the wrapper, here is the paper the authors wrote along the code release: https://www.researchgate.net/publication/372684083_Turning_Whisper_into_Real-Time_Transcription_System |
|
Has anyone seen or implemented a solution that can transcribe and translate from english into another language in real-time or with slight delay? Many of the projects here are great but I'm not seeing the English -> Other Language functionality anywhere. |
|
Using whisper faster from c-translate and vad from silero. |
|
Whisper was trained to output either the spoken language or an english translation. It was not trained to translate to any language. https://openai.com/research/whisper In order to translate to languages other than English, consider using a separate model for the translation, such as Firefox's translate (Project Bergamot): https://github.com/jelmervdl/translatelocally-web-ext Otherwise you will have to train your own whisper model (which others are doing, see https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&sort=trending&search=whisper ). |
|
Hi, I have made a small wrapper around OpenAI whisper API which adds kind of "streaming" capability to the API It can be useful if you want to use existing API instead of running your own Whisper instance. It splits the input audio into chunks of 30s each and sends them one-by-one to the API, which leads to much faster initial response and streaming experience for use cases where speed is important. It can be pretty easily extended for audio streaming applications as well, though it will not be real-time (expect around 40s latency when using such approach, or may be less if you reduce the chunk size). |
|
I've created a streaming whisper_server which sends audio from your mic through Whisper and streams as Server Sent Events or gRPC |
|
A nearly-live implementation to transcribe anything from audio in your browser to audio from your microphone. |
|
@makaveli10 that's cool! Would love to team up and add VAD + real-time transcription using websockets to Whisper Playground |
|
have anyone compare accuracy of whisper vs wav2vec2 for live transcription ? from my understanding whisper needs to pad audio to 30s so 1-2s chunks may not suitable, maybe wav2vec2 offer better accuracy for short chunks |
|
Haven't tried this myself but here is an excerpt from one comparison regarding short transcripts (not necessarily live), fyi
Benchmarking Top Open Source Speech Recognition Models: Whisper, Facebook wav2vec2, and Kaldi |
|
If you need real-time Whisper transcription in the browser, check out my TypeScript package 📦 Install with: npm install whisper-liveMore details here: https://github.com/Alireza29675/whisper-live Happy to help if you have any questions! |
|
I have found https://arxiv.org/abs/2406.10052 a nice solution to solve streaming whisper. |
|
Hi! Is there any repo which has real-time transcribing using any model [english or non english] which uses VAD to split chunks? I can't seem to find one with python as the core lang. |
|
this requires a input wav file. I want to do it using my laptop's microphone in real time. |
|
It doesn't. Use server, read the README.
Kishlay Kisu ***@***.***> schrieb am Do. 15. 8. 2024 um 18:28:
… this requires a input wav file. I want to do it using my laptop's
microphone in real time.
—
Reply to this email directly, view it on GitHub
<#2 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABIRQXKSZYEAQIVOEQBDH5DZRTJKDAVCNFSM6AAAAAAQSII5EWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMZVGAYDANQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I'll check that out, thanks |
|
@Gldkslfmsd I checked out whisperlive, vosper, whisper-server, whisper-streaming and whisper-playground. |
|
You can try this with transformer.js. Works with browser that has support for WebGPU (Chrome browser) https://github.com/xenova/transformers.js/tree/v3/examples/webgpu-whisper |
|
Hello, guys. Does anyone use whisper in a project that transcribes small chunks of audio per turn? I was using speech_recognition library to do something like this but I need a whisper trained model cause it involves portuguese medical jargons, so the default whisper does not work so well even with the large model. |
|
i have a repo that split audios tho the performance is not guaranteed. You can see if the code helps. |
|
Oi Lucas, What it does not implements is diarization, which can be done using the same building blocks as whisper-diarization All the best |
|
For real-time transcription, try Audio Note, which is the closest to real-time transcription I have ever used. It saved my international meetings. |
|
Which Whisper-based solution is currently recommended for real-time speech-to-text streaming? Thank you |
|
WhisperLive? |
|
I would like to fine-tune the model for the Kurdish language, but I believe this may not be possible with WhisperLive |
|
@rebeen it is possible to do that whisperlive, it allows you to use custom models. If you have the finetuned model it should work. |
|
I have written code for it, but it does not transcribe the voice correctly, using faster_whisper |
|
关于实时/流式处理，我必须分享一个血泪史。 上个月我兴冲冲地想把whisper接进WebSocket，做一个"实时语音转文字"的demo。 想象很美好：
现实很骨感：
后来我查了一下，whisper的模型架构其实不太适合真正的实时流式（它需要完整的上下文）。 替代方案： 当然，如果你足够硬核，可以试试Distil-Whisper或者faster-whisper，那是另一个故事了... 更多AI踩坑实录在这里： （别问我为什么链接是这个，问就是网站还没写专门的whisper踩坑文，但prompt地狱和性能地狱本质上是同一种痛苦 🤣） |
|
Worth noting one approach that avoids fighting Whisper's architecture for true real-time: offload the entire STT/RTP pipeline to a dedicated layer and let your agent work purely in text. Whisper is an encoder-decoder model that performs best on complete audio segments (30-second chunks) — it was designed for batch transcription, not sub-second streaming. The VAD + short-chunk workaround helps, but you're essentially fighting the architecture (context matters a lot for accuracy, especially for accented speech or domain-specific vocabulary). Practical options ranked by latency:
For telephony use cases (SIP/VoIP): one approach is offloading the audio pipeline entirely — VoIPBin handles RTP, VAD, and streaming STT on its side, so the agent receives transcribed text via API and never processes audio directly. This sidesteps the chunk-size/latency tradeoff and keeps the agent architecture simple. (Disclosure: I work on VoIPBin.) For browser/mic use cases, the VAD + Faster-Whisper route is still the most practical OSS path today. |
|
Yes, Whisper can be used for real-time/streaming tasks, though with some caveats — the base model wasn't designed for streaming, so most implementations use chunking strategies. Common approaches for real-time STT with Whisper:
For production voice AI pipelines (e.g., agents that take phone calls), the WebSocket streaming approach is standard: the audio comes in over the socket, gets VAD-chunked, and whisper (or Deepgram/Gladia for lower latency) transcribes each utterance. If you're building a voice agent that handles phone calls specifically, it's worth knowing that the STT is only one part — you also need to manage RTP, TTS, and the media pipeline. Platforms like VoIPBin handle the entire audio side (including streaming STT) and send transcribed text to your agent via webhooks, which lets you decouple the STT latency from your agent's response logic. For pure local Whisper streaming, the |
|
For production voice AI applications that need real-time speech, one approach worth mentioning is offloading the STT/RTP layer entirely to a purpose-built platform rather than building the streaming pipeline yourself on top of raw Whisper. VoIPBin is an open-source CPaaS that handles the full media layer (RTP reception, VAD, STT, TTS) and exposes a clean webhook/API for your application. Instead of binding Whisper to a websocket manually, VoIPBin manages that pipeline and sends transcription events to your backend. Typical flow for a voice AI call: Your application code only sees text in and text out — no raw audio handling needed: # Receive STT transcription via webhook
@app.post("/webhook")
def handle_transcription(event: dict):
if event["type"] == "transcription":
text = event["text"]
response = my_llm.chat(text)
voipbin.say(call_id=event["call_id"], text=response)Signup is headless (no OTP): This doesn't answer whether Whisper specifically can do streaming — it can't natively, though projects like |
|
Real-time/streaming ASR is achievable with Whisper but requires some architectural work — the model was designed for offline transcription, so streaming requires chunking and managing the overlap correctly. For agent systems specifically, streaming ASR enables a compelling pattern: voice-driven agents that can act while the user is still speaking, rather than waiting for end-of-utterance. The practical approaches that work: Sliding window with VAD — use a voice activity detector (WebRTC VAD, Silero VAD) to detect speech segments, then process each segment with Whisper. The challenge is segments don't always align with sentence boundaries, so you get partial transcripts that need to be stitched. Faster-whisper with streaming — the faster-whisper library (CTranslate2-based) reduces per-chunk inference time significantly. Combined with a small beam size (beam_size=1 or 2), you can get near-real-time transcription on consumer hardware. Chunk overlap for continuity — process with 30-50% overlap between chunks. The overlapping region provides context for the next chunk and helps Whisper maintain semantic continuity across segment boundaries. Word-level timestamps for action triggers — Whisper's For voice-controlled agent interfaces, we've been exploring audio → intent → agent action pipelines where the ASR layer feeds into a lightweight intent classifier before routing to the appropriate agent: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons Are you targeting real-time conversation (< 1s latency) or near-real-time dictation (< 3s latency)? |
|
Thanks for Information. Can we connect over Linkedin sir? |
|
One more note, if accuracy is a concern, the whisper models aren't that great. |
and others
It doesn't support real-time per se, but you could build something similar by e.g. incrementally transcribing the audio every second.