Satellite entity for Assist pipelines

[synesthesiam]

Date: 2024-07-13
Modified: 2024-08-25

Context

Currently we have "Assist Pipelines" in Home Assistant which represents an assistant to talk to. The devices that talk to these assistants are not represented in Home Assistant. This means that all devices that provide an Assist experience are creating their own entities to make this possible.

We want to standardize these devices to allow a single experience to integrate and configure them in Home Assistant. To do this, we want to propose a new entity component assist_satellite, where each entity will represent a voice satellite. Integrations esphome, voip, and wyoming will implement an assist_satellite platform.

Besides standardization, we also want to remote control the satellite during a setup wizard to confirm that the device is working. To start, we will add the ability to announce a message on the satellite (either text or a media id) and the ability to intercept the next wake word detection (stopping the device from recording/processing a voice command). With these things, a setup wizard can ensure that the device's speaker and microphone are functioning properly.

To start, we will:

• Add an assist_satellite platform with an AssistSatelliteEntity base class
  • The async_accept_pipeline_from_satellite function forwards audio internally to an assist pipeline and manages the entity's state
  • Entity ids for the selected pipeline and VAD sensitivity are available as attributes, and are resolved automatically within
    async_accept_pipeline_from_satellite
  • An abstract on_pipeline_event should be overridden to handle assist pipeline events (stt start, stt end, etc.)
  • The tts_response_finished method must be called when the satellite has finished speaking the text-to-speech response
  • If the ANNOUNCE feature is supported, async_announce must be overridden to handle announcements (media_id will contain the resolved URL)
• Implement the ability to:
  • Announce a message (or media id) with an entity component service call (assist_satellite.announce)
    • entity_id of the assist satellite entity (must support AssistSatelliteEntityFeature.ANNOUNCE)
    • message is the text to announce (only used if media_id is empty
    • media_id is a media id to announce instead of message
    • Attempting to announce while an announcement is already in progress will raise a SatelliteBusyError
  • Intercept the next wake word detection with a websocket command (assist_satellite/intercept_wake_word)
    • entity_id of the assist satellite entity
    • Returns a response of the form { "wake_word_phrase": "<wake word>" } with the detected wake word
    • Attempting to intercept a wake word while an interception is already in progress will raise a SatelliteBusyError

Bigger picture:

• Implement assist satellite for esphome, voip, and wyoming integrations
• Standardize how satellites expose their capabilities (ie, available wake words)
• Standardize how satellites are configured (which wake word to listen to)
• Control the behavior of a satellite from HA during the setup wizard
• Skip wake word and get the spoken text from a command
• Listen for a specific wake word
• Control a satellite from HA using service calls

The states of the AssistSatelliteEntity will be:

• listening_wake_word - device is waiting for the wake word
• listening_command - device is streaming audio with a voice command to HA
• processing - device has stopped streaming audio and is waiting for HA to process the voice command
• responding - device is speaking the response

Disabling the AssistSatelliteEntity will disable the voice features of the device.

Decision

Moving forward with implementation.

Consequences

Users with automations that target existing "Assist in progress" sensors and mute switches will need to be migrated to use the entity instead.

[donburch888]

so...
Will voice satellite and music player still be mutually exclusive, even though both use speakers in the same room ?
Will this allow separating the microphone device (or multiple "ears" an Stuart keeps proposing) component hardware from the speaker component, allowing a media_player to be used ?

[balloob]

That's unrelated. Any integration could offer such an entity that could do all that.

[mvdwetering]

The timer_ringing state seems to be a bit of a weird one.

Does it mean that when state is timer_ringing it can not be listening for wakewords (as that is state idle)?
That would imply there is no way to stop the timer ringing by voice which is unexpected.

Next to that timer_ringing seems unrelated to voice processing. Why does timer ringing impact voice processing?

[balloob]

I think you're right. We should probably add a new "timer ringing" binary sensor. We could even tie that binary sensor as the source of truth for the timer intents that a device supports timers.

[donburch888]

Does it mean that when state is timer_ringing it can not be listening for wakewords

I have only tried a timer once, by adding to wyoming-satellite:
--timer-finished-wav sounds/timer_finished.wav --timer-finished-wav-repeat 10 2

This specifies that when the timer finishes it should play timer_finished.wav 10 times with a 2 second interval between each.
The log shows that it mutes the microphone each time it plays timer_finished.wav...

Jul 24 17:11:06 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:12:00 raspi4 run[2654]: DEBUG:root:Stopped recording to recordings/356900415030859-stt.wav
Jul 24 17:12:00 raspi4 run[2654]: DEBUG:root:Event(type='transcript', data={'text': ' Coughing'}, payload=None)
Jul 24 17:12:00 raspi4 run[2654]: INFO:root:Waiting for wake word
Jul 24 17:12:00 raspi4 run[2654]: DEBUG:root:Started recording to recordings/357420319744407-wake.wav
Jul 24 17:12:00 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:12:00 raspi4 run[9948]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:12:00 raspi4 run[2654]: DEBUG:root:Event(type='synthesize', data={'text': "Sorry, I couldn't understand that", 'voice': {'name': 'en_GB-alan-medium'}}, payload=None)
Jul 24 17:12:01 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:12:01 raspi4 run[9951]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Stopped recording to recordings/357420319744407-wake.wav
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Started recording to recordings/357420319744407-stt.wav
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Detection(name='hey_jarvis_v0.1', timestamp=358910611989624, speaker=None)
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Streaming audio
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Event(type='run-pipeline', data={'start_stage': 'asr', 'end_stage': 'tts', 'restart_on_end': False, 'snd_format': {'rate': 22050, 'width': 2, 'channels': 1}}, payload=None)
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Muting microphone for 0.8995918367346939 second(s)
Jul 24 17:36:50 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:36:50 raspi4 run[10027]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:36:51 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Stopped recording to recordings/357420319744407-stt.wav
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Event(type='transcript', data={'text': ' Set a timer for 5 seconds'}, payload=None)
Jul 24 17:37:03 raspi4 run[2654]: INFO:root:Waiting for wake word
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Started recording to recordings/358923735529097-wake.wav
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:03 raspi4 run[10030]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Event(type='timer-started', data={'id': '01J3HTR50QFWZFM5KPW2XRENTE', 'total_seconds': 5, 'start_seconds': 5}, payload=None)
Jul 24 17:37:03 raspi4 run[2654]: DEBUG:root:Event(type='synthesize', data={'text': 'Timer started', 'voice': {'name': 'en_GB-alan-medium'}}, payload=None)
Jul 24 17:37:04 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:04 raspi4 run[10033]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:08 raspi4 run[2654]: DEBUG:root:Event(type='timer-finished', data={'id': '01J3HTR50QFWZFM5KPW2XRENTE'}, payload=None)
Jul 24 17:37:08 raspi4 run[2654]: DEBUG:root:Muting microphone for 1.3124716553287983 second(s)
Jul 24 17:37:08 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:08 raspi4 run[10035]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:10 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:10 raspi4 run[2654]: DEBUG:root:Muting microphone for 1.3124716553287983 second(s)
Jul 24 17:37:10 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:10 raspi4 run[10037]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:12 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:12 raspi4 run[2654]: DEBUG:root:Muting microphone for 1.3124716553287983 second(s)
Jul 24 17:37:12 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:12 raspi4 run[10039]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:14 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:14 raspi4 run[2654]: DEBUG:root:Muting microphone for 1.3124716553287983 second(s)
Jul 24 17:37:14 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:14 raspi4 run[10041]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:15 raspi4 run[2654]: WARNING:root:Did not receive ping response within timeout
Jul 24 17:37:15 raspi4 run[2654]: DEBUG:root:Server disconnected
Jul 24 17:37:15 raspi4 run[2654]: INFO:root:Disconnected from server
Jul 24 17:37:16 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:16 raspi4 run[2654]: DEBUG:root:Muting microphone for 1.3124716553287983 second(s)
Jul 24 17:37:16 raspi4 run[2654]: DEBUG:root:Connected to snd service
Jul 24 17:37:16 raspi4 run[10043]: Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
Jul 24 17:37:18 raspi4 run[2654]: DEBUG:root:Unmuted microphone
Jul 24 17:37:28 raspi4 run[2654]: DEBUG:root:Connection cancelled: 345718100782196

I guess it makes sense ... when we know we are playing the .wav file, there is no point to check whether it is a wakeword and possibly start a feedback loop. Depending on the number of repeats and delay between each ... yes it is going to be difficult to turn it off by voice.

[mvdwetering]

Ah, so it is a limitation of the current voice satellites. It is about avoiding to listen for wakewords when it is probably going to fail anyway.

I wonder how important it is to have this a very specific state.

If there would be another thing that could interfere with wakeword detection future another state would need to be added because it is not a timer_ringing. Maybe a more generic name would be better/clearer. Something along the lines of wakeword-detection-temporarily-suspended (obviously needs a better name).

btw interesting that I can use voice to stop an alarm/timer going off on my Google Mini. I wonder how they avoided/solved the feedback issue.

[synesthesiam]

This is because I don't have echo cancellation on the Wyoming satellites yet. It's possible, but not part of the tutorial for now.

[synesthesiam]

Implementation: home-assistant/core#123830

[tetele]

Just noticed something about this: what if we have a satellite without wake word, which is not yet triggered? Shouldn't there be a state for that as well? I think idle is that state and there should be a special listening_wakeword state or something.

[synesthesiam]

I'd prefer something like waiting_for_input instead of idle, as the device may be actively doing something. But I don't know if it's really worth differentiating these states. For automations, what really matters is if we're before the voice command, in the voice command, or after the voice command (and speaking or not).

[lgo]

muted - device is muted (in software or hardware)

It would be good to clarify exactly what this means, especially with respect to the other states. For example, I could imagine two different meanings of muted with respect to a satellite device:

• Whether the microphone is muted, which would mean the device technically wouldn't be able to enter the states idle (listening for wake words), listening, or processing but would potentially have responding even while the mic is muted
• Whether the speaker is muted, so the device cannot play back timers, announcements, or other responses triggered external to the satellite device but may be listening (although, listening but not responding is a bit odd so speaker muting feels like it might as well include mic muting)

It sounds like it may be helpful to have a more precise definition for this? (:

Also, I love the tighter integration idea!

[synesthesiam]

I've updated the proposal to remove the muted state. I agree that we need to differentiate between the audio input and output being muted.

[MartinHjelmare]

This proposal has been accepted!

The core PR with the entity implementation is here:
home-assistant/core#125351

[lgo]

This just got released - https://platform.openai.com/docs/guides/realtime

While i's quite a divergence in the paradigm of how currently assist pipelines work (with separate ASR, interpretation, TTS), I figured I'd flag it. It's an interesting alternative where several pipeline steps may happen simultaneously by one source, which significantly reduces the latency. It's been pretty slick to experiment with!