Support for Multimodal Live APIs to achieve Ultra-Low Latency Voice-to-Voice

[BenLad17]

Describe the feature

Currently, the Home Assistant voice assistant pipeline follows a sequential pipeline (STT -> LLM -> TTS). While this architecture is powerful and "local-first" friendly, it introduces significant latency.

1. STT must wait for the end of an utterance (VAD).
2. The LLM then processes the text and generates a response.
3. TTS then generates audio from the text.

In a Voice Satellite setup (e.g., ESP32-S3 devices using ESPHome), this "turn-based" approach creates a gap of several seconds of "dead air," which makes natural conversation difficult.

I propose integrating Multimodal Live APIs, such as Google’s Gemini Multimodal Live, or other low-latency streaming models. Unlike traditional models, these APIs enable continuous, full-duplex audio streaming. Rather than waiting for a sentence to finish, audio is streamed directly from the satellite to the model. The model then streams the audio back while processing it, effectively merging STT, LLM, and TTS into one fluid process.

Key benefits for Home Assistant:

1. Drastic Latency Reduction: By transitioning from a sequential to a streaming model, the "time to first byte" is significantly reduced. The perceived latency shifts from $$L_{\text{total}} = T_{\text{STT}} + T_{\text{LLM}} + T_{\text{TTS}}$$ to a much more responsive $$L_{\text{total}} \approx T_{\text{network roundtrip​​}} + T_{\text{initial processing}}$$.
2. Natural Interaction (Barge-in): Because the stream is bidirectional, a user can interrupt the assistant mid-sentence. The model "hears" the interruption and stops generating immediately, mimicking human conversation.
3. Contextual Awareness: These models can interpret emotional cues and tone of voice, enabling more intuitive smart home control. For example, whispering "Turn off the lights" at night results in a whispered confirmation.

Example commands

see below

Use cases

1. Natural, Fluid Conversation (The "Human" Feel)
   Current voice assistants feel like walkie-talkies. You speak, wait, and then it responds. With a Live API, however, a user can have a back-and-forth conversation about their home.
   Example: "Hey Assist, I'm cooking. What can I make with the ingredients in my smart fridge?"
   [Assist starts listing items...] [Assist starts listing items.] The user interrupts, "Wait, skip the onions; I'm allergic." ... Assist immediately pivots: "Got it, no onions. Let's look at a creamy pasta dish instead."
   Why it needs Live API: Standard pipelines would require the user to wait until the assistant finishes the entire list before giving the "no onions" command.

2. Low-Power Satellite Responsiveness:
   Many users rely on ESP32-based satellites (ESPHome), which have limited processing power.
   Example: A user in the bedroom whispers, "Turn off the hallway lights."
   Why Live API is needed: Rather than the satellite struggling to process a local wake word and waiting for a slow speech-to-text (STT)/text-to-speech (TTS) round trip, the audio is instantly streamed. This low latency enables the lights to turn off almost as quickly as with a physical switch.

3. Contextual and Emotional Awareness
   Multimodal models can "hear" both the environment and the user's tone of voice.
   Example: A user sounds hurried and says, "I'm leaving now!" The assistant recognizes the urgency and automatically triggers the "Away" scene, which locks the doors, arms the alarm, and turns off the iron, without needing a specific list of commands.
   Why it needs the Live API: Traditional speech-to-text (STT) strips away the emotional metadata of the audio. The Live API, however, processes the raw audio, preserving the intent behind the tone.

Anything else?

Maybe worth to look at:

• https://github.com/Steven-Low/gemini_live_hass_integration
• https://github.com/fjfricke/ha-openai-realtime

[kyvaith]

I have implemented a generic Pipecat app for Home Assistant, that integratest with HA MCP. You can use it with realtime Gemini or OpenAI or craft your own pipelinces with other realtime cloud services. You can also design your own flows with n8n-like confgiuration. This is a work in progress, next week I'll release componet for ESPHome to work together (already working, polishing) https://github.com/kyvaith/pipecat-homeassistant