-
Notifications
You must be signed in to change notification settings - Fork 92
Extend TalkSDK and AudioClient with Lemonade ASR+TTS auto-detection #386
Copy link
Copy link
Open
Labels
agentaudioAudio (ASR/TTS) changesAudio (ASR/TTS) changesdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestlemonade 🍋p1medium prioritymedium priorityperformancePerformance-critical changesPerformance-critical changestalkTalk agent changesTalk agent changestrack:consumer-appHermes-competitor consumer product — mobile-first, voice + messaging + memory + skillsHermes-competitor consumer product — mobile-first, voice + messaging + memory + skills
Metadata
Metadata
Assignees
Labels
agentaudioAudio (ASR/TTS) changesAudio (ASR/TTS) changesdomain:multimodalVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAVoice (ASR/TTS), Vision (VLM), Image gen (SD), CUAenhancementNew feature or requestNew feature or requestlemonade 🍋p1medium prioritymedium priorityperformancePerformance-critical changesPerformance-critical changestalkTalk agent changesTalk agent changestrack:consumer-appHermes-competitor consumer product — mobile-first, voice + messaging + memory + skillsHermes-competitor consumer product — mobile-first, voice + messaging + memory + skills
Summary
Wire the new LemonadeASR (#372) and LemonadeTTS (#373) backends into the existing TalkSDK and AudioClient so that
gaia talkseamlessly uses Lemonade server-side audio when available, with graceful fallback to local Whisper/Kokoro.Current State
AudioClient(src/gaia/audio/audio_client.py) hardcodes localWhisperAsrandKokoroTTSTalkSDK(src/gaia/talk/sdk.py) has no backend selection mechanismTalkConfighaswhisper_model_sizebut noasr_backend/tts_backendfieldsChanges Required
src/gaia/talk/sdk.pyasr_backend: str = "auto"andtts_backend: str = "auto"toTalkConfig. "auto" checks Lemonade first, falls back to local. Also accepts "lemonade" or "local".src/gaia/audio/audio_client.py/healthforwebsocket_port→ if present, useLemonadeASR; check loaded models forkokoro-v1→ if present, useLemonadeTTS. Log which backend is active.src/gaia/cli.py--asr-backendand--tts-backendflags togaia talkcommandtests/unit/test_audio_backend_selection.pytests/integration/test_talk_lemonade.pyAuto-Detection Logic
Dependencies
LemonadeASRclass)LemonadeTTSclass)Acceptance Criteria
gaia talkauto-detects and uses Lemonade ASR/TTS when server is running--asr-backendand--tts-backendCLI flags override auto-detectiongaia talkbehavior unchanged when Lemonade is not running