Skip to content

v2.6.0 - Silero VAD, Jetson Support & Parallel Filler

Choose a tag to compare

View all tags
@pjdoland pjdoland released this 23 May 20:46
· 32 commits to main since this release
v2.6.0
4c0be90

New Features

  • Silero VAD replaces WebRTC VAD: The Stage 3 speech-content detector now uses Silero's neural VAD instead of WebRTC. Significantly harder to fool with non-speech sounds (TV chatter, AC hum, distant music) that previously slipped past WebRTC and triggered Whisper hallucinations. Threshold knob is now a probability (0.0-1.0) instead of an aggressiveness level (0-3).

  • Parallel filler + Whisper: The filler phrase no longer waits for the Whisper response before kicking off; both start in parallel as soon as Stages 1-3 validation passes. The filler covers the full transcribe + GPT + TTS latency window instead of just the GPT+TTS portion, which makes the perceived response time noticeably shorter.

  • Layered .env overlays: Settings now compose from three sources (highest precedence first): personalities/{PERSONALITY}/.env → device_overrides/{OUTPUT_DEVICE_TYPE}/.env → root .env. Lets you, e.g., set a different VOICE_GAIN for Squawkers vs Teddy without touching the personality config. Overlay files are .gitignore-d by the existing .env rule. Loaded overlay paths are exposed as settings.LOADED_ENV_OVERLAYS and logged at startup.

  • Jetson deployment support: New docs/JETSON_DEPLOYMENT.md covers running on NVIDIA Jetson hardware. RVC behaves better on memory-constrained devices: CUDA allocator pressure reduced, conversion is retried up to 3 times with backoff on transient CUDA OOM, and RVC is re-warmed before each scheduled event so idle-period cold starts don't stall the first audio chunk.

  • Streaming wake-word inference: wake_word.py now uses streaming inference instead of predict_clip, which lowers per-frame latency. Sub-threshold scores in [0.5, threshold) are logged once per second so you can diagnose flakiness: was the score 0.92 (lower threshold a touch) or 0.50 (mic / audio path problem)?

  • Jarvis personality: New jarvis personality featuring J.A.R.V.I.S. from Iron Man. Uses OpenWakeWord's bundled hey_jarvis pre-trained model (no custom training needed), the fable TTS voice through RVC, and a butler-tone system prompt with 30 in-character filler phrases.

Reliability & Audio Pipeline

  • VOICE_GAIN now applies to RVC-converted audio in addition to the raw TTS path (previously RVC output bypassed the gain stage).
  • Audio recorder is paused during playback to prevent the toy from waking itself with its own voice.
  • Post-playback tail guard added before the mic resumes, so the trailing reverb of the speaker doesn't trigger a new VAD event.
  • Output stream buffer bumped to 4096 frames to eliminate underruns on slower hardware.
  • PortAudio stream is now actually stopped during recorder pause instead of just dropping frames (was leaking CPU on long IDLE stretches).
  • Filler audio is loaded lazily on first use, so startup is faster.
  • Debug audio writes are now gated on SAVE_DEBUG_AUDIO (previously always wrote when debug mode was on).

Documentation

  • CLAUDE.md and README.md brought current with all of the above: Silero VAD references, parallel filler description, RVC tuning knobs (with Jarvis as a worked example), layered env overlays, VAD_THRESHOLD replacing VAD_AGGRESSIVENESS throughout. Personality lists trimmed to the seven actually shipped (fred, jarvis, johnny, kitt, leopold, mr_lincoln, teddy_ruxpin) instead of including .gitignore-d local-only installs.
  • docs/JETSON_DEPLOYMENT.md added.
  • Stray test.py removed from the repo root.
  • SERVICE.md operational notes now gitignored (per-deployment).

Configuration Changes

Breaking (requires .env update on upgrade):

# Old
VAD_AGGRESSIVENESS=2   # 0-3 (WebRTC)

# New
VAD_THRESHOLD=0.5      # 0.0-1.0 probability (Silero); higher = stricter

New optional env vars:

# Layered overlays are automatic; no flag needed. Optional override files:
#   personalities/{PERSONALITY}/.env       # highest precedence
#   device_overrides/{OUTPUT_DEVICE_TYPE}/.env

# Wake-word near-miss logging is automatic; no flag needed (uses existing LOG_LEVEL).

Upgrading:

  1. Update .env: rename VAD_AGGRESSIVENESS=2 → VAD_THRESHOLD=0.5. Tune if needed: bump higher if Whisper still hallucinates on noise; drop lower if valid speech is being rejected.
  2. (Jetson users only) See docs/JETSON_DEPLOYMENT.md for the recommended RVC_DEVICE, allocator, and retry settings.

New dependency

  • silero-vad (replaces webrtcvad)
Assets 2
Loading