Skip to content

Releases: pjdoland/jf-sebastian

v2.7.0 - Spotify Voice Control, Drop-in Devices & GPT-5

Choose a tag to compare

@pjdoland pjdoland released this 22 Jun 12:33

New Features

  • Spotify voice control: Personalities can now control Spotify playback by voice. The model is given nine music_* tools (play, pause, resume, skip, previous, volume, and more), resolves Connect speakers by name (exact, alias, substring, or fuzzy match), and speaks a templated confirmation with no second LLM round-trip. The currently-playing track is injected into conversation context so the character can talk about what is on. Auth uses PKCE (no client secret); scripts/spotify_auth.py and an optional setup.sh walkthrough guide first-time login. Opt in per personality (spotify_enabled) and globally (SPOTIFY_ENABLED). See docs/SPOTIFY_SETUP.md.

  • Drop-in modular output devices: Output devices are now plugin packages that self-register, the same way personalities work. Drop a device package under jf_sebastian/devices/<name>/ and it is auto-discovered; each device owns its own settings and ships its own .env overlay, loaded automatically when that device is selected. A new optional visual seam (requires_visual plus visual_* hooks on the device base class) lets a device drive an on-screen renderer; audio-only devices are unaffected.

  • GPT-5 family by default: The default conversation model is now gpt-5.4-mini, with a new GPT_REASONING_EFFORT knob. The engine handles GPT-5 request semantics (max_completion_tokens, no temperature) and falls back cleanly to gpt-4o-mini for accounts without GPT-5 access.

  • Shimmer input voices for Johnny and Teddy Ruxpin: Both now use the shimmer TTS voice as the RVC input; their distinct character comes from voice conversion.

Conversation & Audio Pipeline

  • System prompt persists across long conversations: The personality's system prompt was stored inside the bounded history deque and got evicted after roughly ten exchanges, causing the character to drift out of character and forget its instructions. It is now pinned outside the history; MAX_HISTORY_LENGTH bounds only the user/assistant turns.
  • Per-personality RVC tuning honored: rvc_filter_radius, rvc_rms_mix_rate, and rvc_protect from personality.yaml are now read and applied (previously silently ignored). RVC is enabled by the presence of a model rather than a redundant flag.
  • Sentence chunker hardened: Extracted into its own module, now abbreviation-aware with a soft length cap, and no longer splits decimals at the streaming edge (so "1.5 ounces" is spoken as words, not "one point five").
  • Debug audio writes are dispatched through a single background writer queue.

Configuration Changes

Audio capture defaults were aligned to the values the project already ships in .env.example, so code, shipped config, and docs now agree:

SAMPLE_RATE=16000                 # was 44100; Silero VAD requires 16000, so the old default silently disabled VAD
SILENCE_TIMEOUT=5.0               # was 10.0
SPEECH_END_SILENCE_SECONDS=1.0    # was 1.5

If you already set these in .env (the setup.sh path does), you are unaffected. If you relied on the bare code defaults, these now take effect.

New optional environment variables:

GPT_MODEL=gpt-5.4-mini            # default; falls back to gpt-4o-mini
GPT_REASONING_EFFORT=low          # GPT-5 family only; empty = model default

SPOTIFY_ENABLED=true              # global gate for the music tools
SPOTIFY_CLIENT_ID=...             # see docs/SPOTIFY_SETUP.md
SPOTIFY_DEFAULT_DEVICE=...        # Connect speaker name
SPOTIFY_DEVICE_ALIASES=...        # name aliases for speaker resolution

Spotify support installs from the new requirements-spotify.txt (spotipy).

Documentation

  • Full fact-check pass across every doc against the source: reconciled the README and Jetson default tables with settings.py, corrected the Johnny/Teddy voice names, the rvc_index_rate (0.5) and rvc_f0_method (harvest) defaults, the wake-word threshold guidance (point at WAKE_WORD_THRESHOLD), and the ARCHITECTURE PPM and eye-control math. Removed references to a nonexistent models/ directory and nonexistent test files.
  • Made explicit that RVC voice models are not distributed with this project; you must train or obtain your own .pth/.index and place it in the personality folder. Without one, the personality uses its raw OpenAI TTS voice.
  • setup.sh adds an optional Spotify install and setup walkthrough and auto-numbers its steps.

Full Changelog: v2.6.0...v2.7.0

v2.6.0 - Silero VAD, Jetson Support & Parallel Filler

Choose a tag to compare

@pjdoland pjdoland released this 23 May 20:46

New Features

  • Silero VAD replaces WebRTC VAD: The Stage 3 speech-content detector now uses Silero's neural VAD instead of WebRTC. Significantly harder to fool with non-speech sounds (TV chatter, AC hum, distant music) that previously slipped past WebRTC and triggered Whisper hallucinations. Threshold knob is now a probability (0.0-1.0) instead of an aggressiveness level (0-3).

  • Parallel filler + Whisper: The filler phrase no longer waits for the Whisper response before kicking off; both start in parallel as soon as Stages 1-3 validation passes. The filler covers the full transcribe + GPT + TTS latency window instead of just the GPT+TTS portion, which makes the perceived response time noticeably shorter.

  • Layered .env overlays: Settings now compose from three sources (highest precedence first): personalities/{PERSONALITY}/.envdevice_overrides/{OUTPUT_DEVICE_TYPE}/.env → root .env. Lets you, e.g., set a different VOICE_GAIN for Squawkers vs Teddy without touching the personality config. Overlay files are .gitignore-d by the existing .env rule. Loaded overlay paths are exposed as settings.LOADED_ENV_OVERLAYS and logged at startup.

  • Jetson deployment support: New docs/JETSON_DEPLOYMENT.md covers running on NVIDIA Jetson hardware. RVC behaves better on memory-constrained devices: CUDA allocator pressure reduced, conversion is retried up to 3 times with backoff on transient CUDA OOM, and RVC is re-warmed before each scheduled event so idle-period cold starts don't stall the first audio chunk.

  • Streaming wake-word inference: wake_word.py now uses streaming inference instead of predict_clip, which lowers per-frame latency. Sub-threshold scores in [0.5, threshold) are logged once per second so you can diagnose flakiness: was the score 0.92 (lower threshold a touch) or 0.50 (mic / audio path problem)?

  • Jarvis personality: New jarvis personality featuring J.A.R.V.I.S. from Iron Man. Uses OpenWakeWord's bundled hey_jarvis pre-trained model (no custom training needed), the fable TTS voice through RVC, and a butler-tone system prompt with 30 in-character filler phrases.

Reliability & Audio Pipeline

  • VOICE_GAIN now applies to RVC-converted audio in addition to the raw TTS path (previously RVC output bypassed the gain stage).
  • Audio recorder is paused during playback to prevent the toy from waking itself with its own voice.
  • Post-playback tail guard added before the mic resumes, so the trailing reverb of the speaker doesn't trigger a new VAD event.
  • Output stream buffer bumped to 4096 frames to eliminate underruns on slower hardware.
  • PortAudio stream is now actually stopped during recorder pause instead of just dropping frames (was leaking CPU on long IDLE stretches).
  • Filler audio is loaded lazily on first use, so startup is faster.
  • Debug audio writes are now gated on SAVE_DEBUG_AUDIO (previously always wrote when debug mode was on).

Documentation

  • CLAUDE.md and README.md brought current with all of the above: Silero VAD references, parallel filler description, RVC tuning knobs (with Jarvis as a worked example), layered env overlays, VAD_THRESHOLD replacing VAD_AGGRESSIVENESS throughout. Personality lists trimmed to the seven actually shipped (fred, jarvis, johnny, kitt, leopold, mr_lincoln, teddy_ruxpin) instead of including .gitignore-d local-only installs.
  • docs/JETSON_DEPLOYMENT.md added.
  • Stray test.py removed from the repo root.
  • SERVICE.md operational notes now gitignored (per-deployment).

Configuration Changes

Breaking (requires .env update on upgrade):

# Old
VAD_AGGRESSIVENESS=2   # 0-3 (WebRTC)

# New
VAD_THRESHOLD=0.5      # 0.0-1.0 probability (Silero); higher = stricter

New optional env vars:

# Layered overlays are automatic; no flag needed. Optional override files:
#   personalities/{PERSONALITY}/.env       # highest precedence
#   device_overrides/{OUTPUT_DEVICE_TYPE}/.env

# Wake-word near-miss logging is automatic; no flag needed (uses existing LOG_LEVEL).

Upgrading:

  1. Update .env: rename VAD_AGGRESSIVENESS=2VAD_THRESHOLD=0.5. Tune if needed: bump higher if Whisper still hallucinates on noise; drop lower if valid speech is being rejected.
  2. (Jetson users only) See docs/JETSON_DEPLOYMENT.md for the recommended RVC_DEVICE, allocator, and retry settings.

New dependency

  • silero-vad (replaces webrtcvad)

v2.5.0 - Pluggable News Headlines

Choose a tag to compare

@pjdoland pjdoland released this 12 May 01:44

New Features

  • Pluggable news headlines provider: Top headlines are now injected into LLM context so personalities can naturally reference current events ("Did you hear about…", "Speaking of the news…"). New NewsProvider interface mirroring the v2.4.0 weather provider pattern with three adapters:

    • RSS (default) — any RSS or Atom feed via feedparser. NPR Topics: News is the built-in fallback if NEWS_RSS_URL is unset, so headlines work out-of-the-box with zero configuration.
    • Hacker News — free public API, parallel fan-out across items.
    • Manual — newline-separated env var for offline / testing (zero network egress).

    Cached 30 min, top 5 headlines per turn. Disable entirely with NEWS_PROVIDER=none.

Improvements

  • Robust RSS handling: HTML markup stripped from titles (e.g., <b>BREAKING:</b> Foo — would otherwise be read aloud by TTS). Polite identifying User-Agent on every HTTP request, since BBC / NYT / Reuters / Cloudflare-fronted feeds frequently 403 the default python-requests/X.Y UA. URL scheme validated (http/https only); allow_redirects=False so a misconfigured feed doesn't silently chain through arbitrary hosts. RequestException cleanly returns None per the ABC contract for uniform negative-caching.
  • Hacker News parallel fetch: top-stories + N item fetches use a shared requests.Session with a ThreadPoolExecutor so cold-miss latency stays bounded even with high NEWS_HEADLINE_LIMIT. Per-item timeout tightened to 2s. Per-item raise_for_status() added (HN's Firebase API can return Cloudflare HTML on 200 during incidents).
  • Better startup observability: news provider init log includes the resolved URL/feed name and a Set NEWS_PROVIDER=none in .env to disable hint, so users can grep one line to see exactly where headlines are coming from. Pre-warm-failure message clarifies that headlines are skipped but the toy itself still works.
  • Documentation refresh: project structure tree in README.md brought current with the actual filesystem (utils modules, scheduler, supervisor scripts, all tests subdirectories, top-level files like ROADMAP.md and CLAUDE.md). README .env Settings now covers Weather Context, News Headlines, Proactive Scheduler, and Supervisor / Watchdog — four whole categories that were previously undocumented. Architecture > Key Components lists the real-world context provider, proactive scheduler, and process supervisor. tests/README.md tree refreshed. docs/CREATING_PERSONALITIES.md now mentions scheduled_events.yaml and RVC models in the personality directory layout. docs/QUICKSTART.md gains a §5b for news with feed table, privacy callout, and child-safety guidance.

Configuration

New env vars (all optional; news is on by default with NPR):

#NEWS_PROVIDER=rss              # or hackernews / manual / none / auto
#NEWS_RSS_URL=https://feeds.npr.org/1001/rss.xml   # NPR by default
#MANUAL_NEWS=Headline one\nHeadline two
#NEWS_HEADLINE_LIMIT=5
#NEWS_CACHE_TTL_MINUTES=30

To disable entirely: NEWS_PROVIDER=none.

Upgrading and don't want news? Add NEWS_PROVIDER=none to your existing .env. Existing installs will start fetching NPR headlines at next restart until that line is added.

Privacy note: the news server (e.g., NPR) sees your IP and access cadence; OpenAI sees the headline text and can infer which feed you chose. Use NEWS_PROVIDER=manual or NEWS_PROVIDER=none for zero third-party news egress.

For child-facing personalities (e.g., teddy_ruxpin, fred): NPR Top News may include violence, politics, or other content unsuitable for children. Consider NEWS_PROVIDER=none or a kid-safe RSS feed when running with these personalities.

New dependency

  • feedparser>=6.0.10 (post-6.0 XXE/billion-laughs hardening enabled by default)

v2.4.1 - Fix Scheduled Events Rejected by State Machine

Choose a tag to compare

@pjdoland pjdoland released this 10 May 12:31

Bug Fixes

  • Scheduled events would never fire: IDLE → SPEAKING was missing from the state machine's VALID_TRANSITIONS table, so try_transition rejected every proactive event the scheduler tried to fire. Symptom in the wild: the scheduler logged Firing scheduled event: …, synthesized TTS via OpenAI (observable in the log), and then silently aborted with no audio playing. Now IDLE → SPEAKING is allowed because scheduled events legitimately bypass the LISTENING/PROCESSING phases — the speech is server-initiated, not user-initiated.
  • Misleading log message for scheduled-event CAS failures: previously logged "lost race to another transition" for any CAS failure (including this hardcoded validation rejection). Now reads "could not enter SPEAKING (state=…)" so the cause is more searchable.

Upgrading

No config changes. Pull, restart, and your personalities/<name>/scheduled_events.yaml events will now actually fire at their scheduled times.

v2.4.0 - Pluggable Weather, Process Supervisor & Proactive Scheduling

Choose a tag to compare

@pjdoland pjdoland released this 09 May 17:12

New Features

  • Pluggable weather/context provider: Refactors the hardwired wttr.in fetch into a WeatherProvider interface with three adapters — wttr.in (default; free, no API key), Home Assistant (local, no third-party egress), and manual (offline / testing, zero network). Auto-selects when WEATHER_PROVIDER is unset; existing ZIPCODE-only setups keep working unchanged. Set WEATHER_PROVIDER=none to disable weather entirely.
  • Process supervisor for unattended deployments: New scripts/supervisor.py wraps python -m jf_sebastian.main with exponential-backoff restart, watchdog kill of hung children via heartbeat-file staleness, enriched crash reports, and permanent-failure detection (CRITICAL log + extended backoff after N consecutive crashes). Includes launchd plist (macOS) and systemd unit (Linux) templates. Designed for museum exhibits, eldercare companions, kids' rooms — anywhere the toy needs to keep running across PortAudio/RVC crashes.
  • Proactive scheduler / ambient mode: Personalities can now define proactive utterances in personalities/<name>/scheduled_events.yaml — morning greetings, bedtime stories, holiday surprises, scheduled reminders. Tiny schedule syntax (HH:MM, HH:MM weekdays, HH:MM YYYY-MM-DD); each event uses say: (verbatim TTS) or prompt: (LLM in character). Events only fire when state is IDLE — never interrupts an in-progress conversation. Quiet-hours suppression with load-time warnings.
  • ROADMAP.md: Public roadmap synthesized from a seven-persona peer-reviewed codebase audit. Documents Tier 1-3 prioritization plus excluded items, with 70 enhancement suggestions evaluated by Borda count.

Bug Fixes

  • Fix urlsplit raising ValueError on a malformed HOME_ASSISTANT_URL and breaking the entire weather pipeline (including auto-fallback to wttr)
  • Fix _refresh_in_flight flag leaking True when the weather provider became unconfigured mid-refresh, suppressing all future refreshes
  • Fix Whisper weather cold-miss (first-conversation timeout caused "I can't check the weather" responses)
  • Fix numpy int16 overflow RuntimeWarning in RMS amplitude calculation
  • Fix race condition in scheduled-event callback that could re-init PyAudio after audio_player.cleanup() during shutdown
  • Fix wake-word detector race during scheduled-event TTS synthesis (added atomic try_transition CAS to state machine)
  • Fix _pause_wake_for_playback flag set even when pause() raised, causing missed resume

Improvements

  • Local-first weather security: Bearer token to Home Assistant is refused over plain HTTP to non-private hosts (loopback, RFC1918, link-local, *.local allowed); URL-quoted entity ID; allow_redirects=False to prevent token leak via cross-host redirects; defensive .get()-chain parsing for wttr; Celsius detection tolerates "C", "celsius", etc.
  • Defense-in-depth supervision: launchd/systemd watch the supervisor; supervisor watches the child; heartbeat thread (utils/heartbeat.py) reports liveness so even hung-PROCESSING is detected and killed via os.killpg on the child's process group (kills ffmpeg / RVC subprocesses, not just the immediate child)
  • Crash report enrichment: PID, personality, ran_for, heartbeat age at exit, hostname, Python version, last 100 log lines; pruned to most recent N (default 200) so disk can't fill on a permanent-failure loop
  • Rotating logs: jf_sebastian.log and supervisor.log use RotatingFileHandler (10MB × 5) so unattended deployments can't fill disk
  • Atomic state transitions: New StateMachine.try_transition(expected, target, trigger) compare-and-swap closes TOCTOU windows for any caller that needs to gate on current state; try_transition and transition_to share a single body via _apply_transition_locked
  • Cooperative shutdown: SIGTERM/SIGINT use cooperative _running = False instead of sys.exit() from a signal handler; SystemExit(0) from handler when SIGTERM arrives during __init__ so the supervisor doesn't have to escalate to SIGKILL
  • Hobbyist-friendly schedule UX: weekdays / weekends aliases, friendlier error messages for typos ("empty weekday token" instead of cryptic unknown weekday ''), past-dated one-shot warnings at load, items listed in scheduler startup log so users can verify their event loaded
  • Updated wake word models for Fred, Johnny, KITT, Leopold
  • Documentation: New "Running Unattended" section in README; QUICKSTART covers all three weather providers + scheduled events; ARCHITECTURE diagram now shows supervisor + heartbeat layers; CLAUDE.md catalogs the new modules

Configuration

New env vars (all optional, sensible defaults):

# Weather provider (auto-selects from configured vars if unset)
#WEATHER_PROVIDER=wttr             # or homeassistant / manual / none / auto
#HOME_ASSISTANT_URL=http://homeassistant.local:8123
#HOME_ASSISTANT_TOKEN=...
#HOME_ASSISTANT_WEATHER_ENTITY=weather.home
#MANUAL_WEATHER=Sunny and 72F

# Proactive scheduler (per-personality scheduled_events.yaml)
SCHEDULER_ENABLED=true
#QUIET_HOURS_START=22:00
#QUIET_HOURS_END=07:00

# Supervisor / watchdog (only relevant when running scripts/supervisor.py)
#HEARTBEAT_FILE=/tmp/jf_sebastian.heartbeat
#WATCHDOG_TIMEOUT=60.0
#RESTART_BACKOFF_INITIAL=1.0
#RESTART_BACKOFF_MAX=60.0
#CRASH_REPORT_DIR=./crash_reports/

See .env.example for the full annotated list.

v2.3.0 - Real-World Context, Cross-Platform & Stability

Choose a tag to compare

@pjdoland pjdoland released this 25 Mar 18:43

New Features

  • Real-world context for conversations: LLM now receives current date/time and weather (via wttr.in) so personalities can naturally answer questions like "What time is it?" — configure with ZIPCODE in .env
  • Headless output device: New headless device type for computer-only playback without hardware
  • GPT-5 model compatibility: Automatic parameter adaptation for GPT-5 models (token limits, temperature constraints)
  • Multi-stage silence detection: 4-stage audio validation pipeline prevents Whisper hallucinations and unnecessary API calls
  • Cross-platform support: Linux/Jetson compatibility for audio, GPU detection, and setup
  • Fred (Mister Rogers) personality: New personality added

Bug Fixes

  • Fix race condition in audio recorder when stop_recording called from callback thread
  • Fix race condition in audio recorder continuous mode
  • Fix RVC crash when vc_single returns tuple with None audio
  • Fix RVC audio playing too fast by propagating actual sample rate
  • Fix echo suppression getting stuck when skipping LISTENING state

Improvements

  • Codebase cleanup: Fixed bugs, deduplicated code (Squawkers as HeadlessDevice subclass, AudioPlayer helper extraction), removed dead code (unused mock classes, commented-out blocks)
  • Efficiency: Cached pyphen dictionary in PPMGenerator, capped state machine transition history
  • RVC reliability: Device validation, fail-fast on permanent errors, Jetson-specific installation support
  • Updated wake word models for fred, johnny, kitt, leopold, mr_lincoln, teddy_ruxpin
  • Added CLAUDE.md for Claude Code guidance

v2.2.0 - Teddy Ruxpin Personality & Audio Optimizations

Choose a tag to compare

@pjdoland pjdoland released this 16 Jan 14:24

New Features

Teddy Ruxpin Personality

  • Added complete Teddy Ruxpin personality (classic 1980s storytelling bear from Grundo)
    • Character with friends Grubby, Princess Aruzia, and Newton Gimmick
    • 30 adventure-themed filler phrases about crystals, treasures, and exploring
    • Wake word: "Hey, Teddy Ruxpin"
    • Warm, friendly Echo voice with childlike wonder
    • RVC voice conversion support for authentic Teddy Ruxpin voice

Performance Improvements

Audio Processing Optimizations

  • Reduced macOS audio stream delay from 2.0s to 0.5s
    • Saves 1.5 seconds between filler and response playback
    • Significantly improves conversation flow

RVC Processing Optimizations

  • Optimized RVC settings for 30-40% faster voice conversion:
    • Disabled pitch median filtering (filter_radius: 0)
    • Reduced volume envelope mixing (rms_mix_rate: 0.1)
    • Reduced consonant protection (protect: 0.2)
  • Applied to K.I.T.T. and Teddy Ruxpin personalities
  • Combined optimizations save approximately 3.8 seconds per interaction

Code Organization

Repository Improvements

  • Moved RVC optimization script to scripts/benchmark_rvc.py
  • Cleaned up personality documentation (removed redundant READMEs)
  • Updated personalities README with Teddy Ruxpin configuration examples

Technical Details

The RVC optimizations maintain voice quality while significantly improving processing speed. The settings are tuned for the pm (Parselmouth) pitch detection method which offers the best balance of speed and quality for real-time animatronic applications.

v2.1.1 - Improved RMS Detection

Choose a tag to compare

@pjdoland pjdoland released this 02 Jan 15:39

Improvements

Better Silence Detection

  • Peak RMS Detection: Switched from average RMS to peak RMS using 100ms sliding windows
    • Prevents silence from dragging down amplitude measurements
    • Detects speech even when surrounded by quiet audio
    • More accurate filtering of background noise

Enhanced Logging

  • Added detailed RMS logging with 📊 indicator for easier threshold tuning
  • Shows both peak RMS value and current threshold for each audio capture

Configuration Updates

  • Updated MIN_AUDIO_RMS threshold from 800 to 60 (appropriate for peak RMS measurements)
  • Removed all hard-coded RMS values - now uses settings.MIN_AUDIO_RMS throughout

Transcription Filtering

  • Added "bye" to meaningless transcription filter to reduce false positives
  • Reduces erroneous wake-up triggers from background noise

Technical Details

The new peak RMS algorithm:

  1. Divides audio into 100ms windows with 50% overlap
  2. Calculates RMS for each window
  3. Returns the maximum RMS value found

This approach is much more effective at detecting actual speech in buffers that contain both speech and silence, significantly improving the accuracy of the silence filtering system.

v2.1.0 - Gapless Audio & Continuous Conversation

Choose a tag to compare

@pjdoland pjdoland released this 02 Jan 14:28

Major Features

🎵 Gapless Audio Playback

  • Implemented persistent stream sessions for seamless multi-chunk audio
  • Filler audio flows directly into response chunks with no gaps
  • Works with both Teddy Ruxpin (44.1kHz) and Squawkers McCaw (48kHz)

💬 Continuous Conversation Mode

  • Multi-turn dialogues without requiring wake word for each turn
  • Natural back-and-forth conversation flow
  • Configurable conversation timeout

RVC Warmup

  • Pre-loads RVC models during startup
  • Eliminates first-use delay for voice conversion
  • Faster initial responses

🚗 New Kitt Personality

  • K.I.T.T. from Knight Rider with RVC voice conversion
  • Custom voice model integration
  • Complete with character-appropriate responses

⚙️ Configurable Filler Audio

  • New ENABLE_FILLER_AUDIO environment variable
  • Option to disable filler phrases for faster responses
  • Maintains backwards compatibility

Improvements

  • macOS Audio Fixes: Resolved audio playback hangs and stream timeout issues
  • RVC Index Feature: Enabled index_rate for better voice quality (fixed faiss/PyTorch OpenMP conflict)
  • Transcript Validation: Prevents responses to empty or meaningless speech (um, uh, etc.)
  • Audio Recorder: Enhanced reliability for continuous conversation mode
  • Documentation: Updated setup and configuration guides

Bug Fixes

  • Fixed CoreAudio stream abandonment strategy on macOS
  • Resolved audio recorder restart issues in continuous mode
  • Fixed multiple playback timing and buffer management issues
  • Corrected buffer drain timing to prevent stream close timeouts

Technical Details

New APIs:

  • AudioPlayer.start_playback_session()
  • AudioPlayer.write_session_chunk()
  • AudioPlayer.end_playback_session()

Backwards Compatibility:

  • All changes are backwards compatible
  • Existing play_stereo() API maintained for single-shot playback
  • No breaking changes to configuration or personality files

Full Changelog: v2.0.0...v2.1.0

v2.0.0 - Major Setup & Stability Improvements

Choose a tag to compare

@pjdoland pjdoland released this 23 Dec 21:15

J.F. Sebastian v2.0.0

Major release with improved setup experience, Python 3.10 enforcement for RVC compatibility, enhanced audio playback stability, and new personality support.

🚀 Setup & Installation Improvements

Python 3.10 Enforcement & Auto-Installation

  • Auto-detect Python 3.10: Automatically finds and uses Python 3.10 if available on system
  • Auto-install Python 3.10: Offers to install via pyenv or Homebrew if not found
  • Smart venv management: Detects and offers to recreate venv if using wrong Python version
  • RVC compatibility: Enforces Python 3.10.x specifically (3.11+ not compatible with RVC)
  • Version file: Added .python-version for automatic version selection with pyenv

Integrated RVC Installation

  • One-step setup: RVC installation now integrated into main setup script
  • Automatic pip management: Handles pip downgrade to 24.0, RVC installation, and pip upgrade automatically
  • Interactive prompts: Optional RVC installation during setup (defaults to No)
  • Verification testing: Automatically tests RVC import after installation

Enhanced Setup Experience

  • Optional filler generation: Filler audio generation now optional during setup (saves 2-3 minutes)
  • Improved error messages: Clear guidance when Python version requirements not met
  • Better documentation: Comprehensive RVC installation guide with troubleshooting

🎯 Audio Playback Stability

Sequential Queue Architecture

  • Eliminated race conditions: Restructured audio system to use single sequential playback queue
  • Seamless transitions: Filler audio flows directly into response chunks without gaps
  • Interruptible playback: Audio now writes in chunks, allowing graceful interruption
  • No more hangs: Fixed blocking PyAudio calls that prevented playback interruption

State Management Improvements

  • Prevent premature transitions: System now waits for all audio chunks to complete before transitioning to IDLE
  • Recovery system fixes: Recovery timeout disabled during active sequential playback
  • Extended timeouts: Increased PROCESSING timeout from 15s to 30s to accommodate normal processing
  • Better state tracking: Added _sequential_playback_active flag for proper state coordination

Performance Optimizations

  • Pre-loaded filler audio: All filler audio loaded at startup (eliminates 14s conversation pauses)
  • Faster audio loading: Switched to soundfile library for faster WAV parsing
  • Capture post-wake audio: Records audio immediately after wake word to prevent missing first words
  • Optimized transitions: Seamless filler-to-response transitions without delays

🤖 New Features

K.I.T.T. Personality

  • Knight Rider's AI: Added K.I.T.T. personality from the classic TV series
  • Custom wake word: Trained "Hey Kitt" wake word model with 90s F1 score
  • Optimized speech: K.I.T.T. speech rate adjusted to 0.92 for better character accuracy

RVC Voice Conversion Support

  • Custom voices: Full support for Retrieval-based Voice Conversion (RVC)
  • Streaming compatible: RVC integrated with streaming TTS pipeline
  • Comprehensive docs: Added detailed RVC setup and configuration guide
  • Pip compatibility workaround: Documented and automated pip 24.0 workaround for RVC dependencies

🏗️ Architecture Improvements

Modular Device System

  • Device architecture: Refactored to support multiple device types (Teddy Ruxpin, Squawkers McCaw)
  • Device-specific audio: Each device can have custom filler audio and PPM configurations
  • Extensible design: Easy to add new animatronic devices

Wake Word Improvements

  • Better detection: Fixed wake word detection during active interactions
  • Improved models: Updated wake word models with 90s+ F1 scores
  • Leopold model update: Refreshed Leopold personality wake word

🐛 Bug Fixes

  • Fixed AttributeError in state machine (use state instead of current_state)
  • Fixed filler playback timeout and flag management issues
  • Fixed missing import for find_audio_device_by_name
  • Resolved git tracking issues with personality configurations
  • Fixed wake word filename references after model updates

📦 Configuration Changes

  • El Rey personality: Excluded from version control (private personality)
  • Updated configs: Refreshed personality configurations for Leopold and K.I.T.T.
  • Python version: All documentation updated to specify Python 3.10.x requirement

Breaking Changes

  • Python 3.10 required: Python 3.11+ no longer supported due to RVC dependency requirements
  • Setup script changes: Setup now includes RVC installation step (step 6/12)
  • Filler generation: Now optional during setup (was previously automatic)

Upgrade Notes

If upgrading from v1.0.0:

  1. Ensure you're using Python 3.10.x (setup script will help with this)
  2. Re-run ./setup.sh to recreate venv with correct Python version if needed
  3. RVC users: Setup will offer to install RVC dependencies automatically
  4. Regenerate filler audio if you skipped during setup: python scripts/generate_fillers.py