Skip to content

Releases

RealtimeVAD 0.1.1

Choose a tag to compare

@KoljaB KoljaB released this 14 Nov 21:38

Quiet streams: any number of extra quiet detectors on one VAD, each with its own sensitivity. Nothing changes for existing code: without a stream the events are the same as in 0.1.0.

pip install -U realtimevad
vad = VAD()
cut = vad.add_stream("cut", offset_db=6.0, quiet_s=0.3)   # quiet 6 dB sooner than pause, held for 0.3 s
for event in vad.feed(chunk):
    if event.kind == "quiet" and event.stream == "cut":
        ...                                               # event.quiet.start, .duration
    elif event.kind == "sound" and event.stream == "cut":
        ...                                               # event.quiet.start, .end, .duration
cut.offset_db = 9.0                                       # any setting, any time

New

  • Quiet streams: vad.add_stream(name, offset_db=..., quiet_s=..., rearm_s=...) adds one more quiet detector to the same VAD: its own offset on the quiet threshold pause uses, its own minimum length and its own rule for what ends the quiet. A stream reports quiet and sound events with where the audio went quiet and for how long. It reuses the instance's levels and noise tracking, so Silero runs once. start, pause, resume and end are identical with and without streams (20 minutes of meeting audio, event by event). Four streams cost about 0.5 ms per second of audio. README, design note.
  • block_events=True: a block event for every 10 ms block with its level, the quiet threshold and over_db, how far the block is from counting as quiet, for callers who want their own rule.
  • benchmarks/quiet_streams.py checks the equality and measures the cost on your own WAVs.

RealtimeVAD 0.1.0

Choose a tag to compare

@KoljaB KoljaB released this 13 Nov 22:54

The product is the early pause: a median 92 ms before Silero notices that the speaker stopped. end is now exactly Silero's end, and pause got more accurate at the cost of about 8 ms of lead.

pip install -U realtimevad

Numbers and tables: README, docs/results.md.

Changed behaviour

  • end comes where Silero alone ends the segment. Up to 0.0.5 the quiet clock could end a segment earlier. That is now the option VAD(early_end=True) (also on VADIterator); Segment.early says whether the quiet clock made the end. If you relied on the earlier end, set it.
  • pause is repeated. When sound comes back after a pause and the audio goes quiet again, a new pause follows. The last pause is more than 100 ms too early in 1 to 9 % of clips instead of 6 to 11 %; its median lead over Silero is 92 ms instead of 100 ms on the one-speaker turns. VAD(repause_s=None) gives one pause per stretch of speech, as before. Sound in the trailing silence does not repeat a pause.

New

  • A short first word is kept with a long start_s. With start_s raised to reject clicks (the demo uses 0.32 s), a short "Hey" before more speech did not last long enough and the segment began at the second word. The start now reaches back to speech of 64 ms or more that stopped less than end_s before. A short sound that stands alone still starts nothing. Nothing changes with the default start_s.
  • pause_mode: "accurate" (default), "balanced" or "fast". "fast" leads Silero by 118 to 130 ms and puts about one in five last pauses inside the last word.
  • resume event (VAD(resume_events=True)): the speech went on after a pause; cancel what you started.
  • Steady noise close under the speaker is subtracted before the quiet check (noise_subtract_db). A noisy quiet line and a quiet phone line had no lead in 0.0.5 and now lead by 84 to 114 ms (simulated noise and line).
  • LiveKit plugin, now on PyPI: pip install livekit-plugins-realtimevad (0.1.0). realtimevad.VAD.load() in place of silero.VAD.load(), and SpeculativeSTT starts LiveKit's preemptive generation at pause. A community plugin, not made by LiveKit. See its README.

Fixed

  • A DC offset in the input no longer hides the silence from the quiet clock (levels are measured behind a 20 Hz high-pass).
  • The "Silero ignores speech-loud audio" recovery could restore a state from inside the previous segment and report a phantom segment.
  • With the early end, the recovery could run on other frames than without it, and one clip of 7,080 ended later than Silero alone.
  • With the early end, a segment that opened right after an early end could end up to three frames before Silero's own count.
  • Demo page: the waveform was drawn later than the detector's events by the time the connection took to open, so a pause could look as if it cut into the speech.

Tools and demo

  • benchmarks/long_stream.py: turns in one long stream against a fresh detector per turn. No drift on 72 minutes of one-speaker turns (results); a test now guards it.
  • python demo/server.py --record DIR keeps a demo session's audio and events in local files, so a session that looked wrong can be replayed.
  • The demo page now shows the last pause against Silero as its headline, and the demo video was recorded again with 0.1.0.

Removed

  • No Pipecat integration is planned: its speech recognizers finalize only when the VAD reports the stop, so nothing can start at pause (issue 4).

Known

  • A reported drift of pause in long sessions did not reproduce: 72 minutes as one stream gave the same pauses as a fresh detector per turn.
  • benchmarks/long_stream.py shows no lead for about half the turns when it is given a recorded demo session. That is a weakness of the tool, not of the detector.

Limits

Alpha. The lead needs a close microphone; on a distant one there is none, and RealtimeVAD then behaves like Silero alone. See the README.

Files

File SHA-256
realtimevad-0.1.0-py3-none-any.whl 7b3123df0925c174c2b0e725cb00f0f825ddb05a5335cddb557682000df84350
realtimevad-0.1.0.tar.gz 3e4fbb31aea6876c0541958fa03a7339ba664f4b8e3c239eca01071f47799344

RealtimeVAD 0.0.5

Choose a tag to compare

@KoljaB KoljaB released this 12 Nov 19:31

Quiet speakers are no longer cut, short "mm-hmm"s are heard, and the end now beats Silero's in 82 % of turns instead of 51 %.

pip install -U realtimevad

Fixed

  • Soft word endings cut on quiet input (#1). "Quiet" had a fixed floor at about −54 dBFS. For a quiet speaker (a phone line at source level) that floor sat a few dB under the speech, and the segment ended while they were still talking. The floor now follows the speaker: quiet_rms or 20 dB below their own level, whichever is lower (floor_margin_db). On 3,401 real turns at −52 dBFS, turns that lost 100 ms of speech or more went from 868 to 26 (Silero alone: 23).
  • Short backchannels not reported as speech (#2). Silero scores many short hums between 0.5 and 0.9, below the start threshold. Such speech now starts a segment after about 0.13 s at 0.5, if it is clearly louder than the room (soft_start_s). Of 200 real backchannels, 175 are reported instead of 150; Silero itself stays below 0.5 for 15.
  • A click in the quiet after a word stopped the early end. One loud 10 ms block (a click, a lip smack, a breath) left the end to Silero. The quiet count now starts again after it. end is earlier than Silero alone in 82 % of turns (was 51 %), by a median 70 ms (was 18 ms).
  • "Never later than Silero alone" is exact now. Silero's state recovery comes on the same frame with and without the quiet clock.

New

  • normalize_to_dbfs (off by default): amplifies what Silero sees for quiet or phone audio. At −62 dBFS, turns that lost 100 ms or more went from 419 to 40. It has a small cost at normal level, see the README.
  • A LiveKit Agents plugin in the repository (not on PyPI): realtimevad.VAD.load() in place of silero.VAD.load(), with measurements against the Silero plugin on the same audio.
  • An edge-case test suite on about 100 MB of freely licensed audio (8 languages, noise, music, coughs, whisper, real backchannels), downloaded by script, not in git.
  • benchmarks/quiet_input.py to reproduce the quiet-input measurement on your own recordings.

Changed numbers

  • Speech-off lead: a median 100 ms before Silero's probability drops (was 110 ms), 206 ms before VADIterator's end, on 3,279 real turns. The 10 ms went into not cutting quiet speakers.
  • On a noisy quiet line (noise within about 20 dB of the speaker) there is no early end any more. It ends where Silero does.

Behaviour changes to know

  • More speech starts a segment (the soft start). Turn it off with VAD(soft_start_s=None).
  • The old fixed floor is VAD(floor_margin_db=None).

Limits

Alpha. The numbers come from one speaker and one capture chain; quiet and phone input were simulated from those recordings. See Quiet input and phone audio and Limits.

Files

File SHA-256
realtimevad-0.0.5-py3-none-any.whl a66a0731c0da8be31e8e47065f6d37e2ff706629b018575469bf5f77722f92a9
realtimevad-0.0.5.tar.gz f8babee8818932c150f46469cd4635616ce8abe0eede2427c0cf7891c19308ab

RealtimeVAD 0.0.4

Choose a tag to compare

@KoljaB KoljaB released this 12 Nov 00:00

First public release of RealtimeVAD: know that the speaker stopped, about 110 ms before Silero does.

pip install realtimevad

What it does

A voice assistant can't do anything until it knows the user has stopped talking. RealtimeVAD keeps Silero VAD to decide what is speech and adds a faster check: every 10 ms it compares the audio level with the room's noise. The moment the audio goes quiet, you get a pause event and can start speculative work (ASR finalization, the LLM call, TTS), then cancel it if speech continues.

  • Lead: speech-off a median 110 ms before Silero's probability drops, and 214 ms before Silero's VADIterator reports the end, measured on 3,238 real voice-assistant turns.
  • Price: it also fires in short gaps between words, about ten times as often as Silero. Use pause only for work you can cancel.
  • Events: start, pause, end, and optional onset.
  • Drop-in: from RealtimeVAD import VADIterator replaces Silero's VADIterator with the same constructor and results.
  • Runs on: one CPU thread. Needs only numpy and onnxruntime; the Silero model is bundled, no torch.

Also in the repository

  • A live demo that shows, on your own microphone, how much sooner RealtimeVAD knew you stopped than Silero alone.
  • SPEC.md with what the detector guarantees.
  • benchmarks/speech_off_lead.py to reproduce the measurement on your own recordings.

Status

Alpha. The API may still change. The numbers come from one speaker and one capture chain with noise suppression; see Limits.

Files

File SHA-256
realtimevad-0.0.4-py3-none-any.whl 8547c743b735a3a9e8aa80b6461a35f8dc85824898fc515d074cd7dc2bce8fbf
realtimevad-0.0.4.tar.gz c580f9742ec2d5fde083308c9e3235da2ab0a40f2edc521e1fdf8e4fc1a2a9cc