Skip to content

RealtimeVAD 0.1.0

Choose a tag to compare

@KoljaB KoljaB released this 13 Nov 22:54
· 2 commits to main since this release

The product is the early pause: a median 92 ms before Silero notices that the speaker stopped. end is now exactly Silero's end, and pause got more accurate at the cost of about 8 ms of lead.

pip install -U realtimevad

Numbers and tables: README, docs/results.md.

Changed behaviour

  • end comes where Silero alone ends the segment. Up to 0.0.5 the quiet clock could end a segment earlier. That is now the option VAD(early_end=True) (also on VADIterator); Segment.early says whether the quiet clock made the end. If you relied on the earlier end, set it.
  • pause is repeated. When sound comes back after a pause and the audio goes quiet again, a new pause follows. The last pause is more than 100 ms too early in 1 to 9 % of clips instead of 6 to 11 %; its median lead over Silero is 92 ms instead of 100 ms on the one-speaker turns. VAD(repause_s=None) gives one pause per stretch of speech, as before. Sound in the trailing silence does not repeat a pause.

New

  • A short first word is kept with a long start_s. With start_s raised to reject clicks (the demo uses 0.32 s), a short "Hey" before more speech did not last long enough and the segment began at the second word. The start now reaches back to speech of 64 ms or more that stopped less than end_s before. A short sound that stands alone still starts nothing. Nothing changes with the default start_s.
  • pause_mode: "accurate" (default), "balanced" or "fast". "fast" leads Silero by 118 to 130 ms and puts about one in five last pauses inside the last word.
  • resume event (VAD(resume_events=True)): the speech went on after a pause; cancel what you started.
  • Steady noise close under the speaker is subtracted before the quiet check (noise_subtract_db). A noisy quiet line and a quiet phone line had no lead in 0.0.5 and now lead by 84 to 114 ms (simulated noise and line).
  • LiveKit plugin, now on PyPI: pip install livekit-plugins-realtimevad (0.1.0). realtimevad.VAD.load() in place of silero.VAD.load(), and SpeculativeSTT starts LiveKit's preemptive generation at pause. A community plugin, not made by LiveKit. See its README.

Fixed

  • A DC offset in the input no longer hides the silence from the quiet clock (levels are measured behind a 20 Hz high-pass).
  • The "Silero ignores speech-loud audio" recovery could restore a state from inside the previous segment and report a phantom segment.
  • With the early end, the recovery could run on other frames than without it, and one clip of 7,080 ended later than Silero alone.
  • With the early end, a segment that opened right after an early end could end up to three frames before Silero's own count.
  • Demo page: the waveform was drawn later than the detector's events by the time the connection took to open, so a pause could look as if it cut into the speech.

Tools and demo

  • benchmarks/long_stream.py: turns in one long stream against a fresh detector per turn. No drift on 72 minutes of one-speaker turns (results); a test now guards it.
  • python demo/server.py --record DIR keeps a demo session's audio and events in local files, so a session that looked wrong can be replayed.
  • The demo page now shows the last pause against Silero as its headline, and the demo video was recorded again with 0.1.0.

Removed

  • No Pipecat integration is planned: its speech recognizers finalize only when the VAD reports the stop, so nothing can start at pause (issue 4).

Known

  • A reported drift of pause in long sessions did not reproduce: 72 minutes as one stream gave the same pauses as a fresh detector per turn.
  • benchmarks/long_stream.py shows no lead for about half the turns when it is given a recorded demo session. That is a weakness of the tool, not of the detector.

Limits

Alpha. The lead needs a close microphone; on a distant one there is none, and RealtimeVAD then behaves like Silero alone. See the README.

Files

File SHA-256
realtimevad-0.1.0-py3-none-any.whl 7b3123df0925c174c2b0e725cb00f0f825ddb05a5335cddb557682000df84350
realtimevad-0.1.0.tar.gz 3e4fbb31aea6876c0541958fa03a7339ba664f4b8e3c239eca01071f47799344