Wyoming ignores second assistant's wake word. #101942

HarvsG · 2023-10-13T14:30:44Z

The problem

I have two voice assistants - a completely HA/NabuCasa assistant that uses 'hey rhasspy' and a ChatGPT/NabuCasas voice assistant that is 'hey jarvis'.
My hope is to ask rhasspy to control the house and Jarvis when I want facts and information.
I run a home-assistant satellite on raspberry pi 3.

Whichever of the two is favourited works well. However if I make a 'hey jarvis' command whilst 'hey rhasspy' is favourited 'jarvis' ignores me. And vice versa if I switch the favourites.

What version of Home Assistant Core has the issue?

2023.10.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Wyoming

Link to integration documentation on our website

https://www.home-assistant.io/integrations/wyoming

Diagnostics information

No diagnostics information

No additional log output when I enable debugging

Example YAML snippet

No YAML

Anything in the logs that might be useful for us?

Logger: homeassistant.components.wyoming.wake_word
Source: components/wyoming/wake_word.py:123
Integration: Wyoming Protocol (documentation, issues)
First occurred: 2:53:34 PM (2 occurrences)
Last logged: 2:53:34 PM

Expected wake word hey_rhasspy_v0.1 but got hey_jarvis_v0.1, skipping

Additional information

I think the wyoming may be the route cause of these issues I believed to be to do with OpenWakeWord and home-assistant satellite.

home-assistant/addons#3259 see in this issue that the wake word is detected by OpenWakeWord, however the trigger information in the log is duplicated several times.
synesthesiam/homeassistant-satellite#13

home-assistant · 2023-10-13T14:30:52Z

Hey there @balloob, @synesthesiam, mind taking a look at this issue as it has been labeled with an integration (wyoming) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of wyoming can trigger bot actions by commenting:

@home-assistant close Closes the issue.
@home-assistant rename Awesome new title Renames the issue.
@home-assistant reopen Reopen the issue.
@home-assistant unassign wyoming Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

wyoming documentation
wyoming source
_{^{(message by IssueLinks)}}

HarvsG · 2023-10-13T14:45:48Z

Ok, after some digging, I can refine the behaviour.
If you restart OpenWakeWord it will only detect the wake-word currently favourited assistant.
If you switch favourite assistant whilst OpenWakeWord is running, it will load the new wake word. It will still detect the previous favourite (with the duplications in the debug log) but wyoming will skip/ignore the old favourite wake word.

So it seems multiple assistants are not supported in wyoming, and they can be supported in OpenWakeWord via this work-around of favourite-switching

I will also post this to home-assistant/addons#3259 as I think it clarifies some behaviour.

synesthesiam · 2023-10-13T17:13:35Z

homeassistant-satellite only runs one pipeline at the moment (you can change it with --pipeline <NAME>)
I may extend this in the future so multiple pipelines can be fed the same audio data simultaneously.

HarvsG · 2023-10-15T11:12:01Z

homeassistant-satellite only runs one pipeline at the moment (you can change it with --pipeline <NAME>) I may extend this in the future so multiple pipelines can be fed the same audio data simultaneously.

That may well be true, but I don't think it's the (only) 'issue' here, as as the error demonstrates, the second wake work does reach Wyoming, but is ignored by it.

ThreepE0 · 2023-10-16T02:42:47Z

homeassistant-satellite only runs one pipeline at the moment (you can change it with --pipeline <NAME>) I may extend this in the future so multiple pipelines can be fed the same audio data simultaneously.

That may well be true, but I don't think it's the (only) 'issue' here, as as the error demonstrates, the second wake work does reach Wyoming, but is ignored by it.

It’s ignored because it doesn’t match the pipeline it’s pointing to

HarvsG · 2023-10-16T09:58:16Z

Ok, but I don't see how home assistant satellite could or should knows which pipeline(s) it is supporting - it's just sending audio data back and forth - surely it could only be told by wyoming. Why can't home assistant satellite and Openwakeword be pipeline agnostic and just let Wyoming decide which wake word corresponds to which pipeline? (Unless there's a desire for different devices to have different pipelines with the same wake-word - in which case Wyoming could still make this decision based on source device meta-data)
We are straying into architectural discussions here, which makes this more than a feature request than an issue.

If what you are saying is that decisions have already been made that supported pipeline(s) should be specified in homeassistant satellite or the wake-word engine and that Wyoming would work correctly if they supported multiple pipelines then this isn't a Wyoming issue and I'll close it.

If a decision hasn't been made then it seems,
architecturally to be most efficient to have the computationally expensive stuff - speech and wake word detection - done once and agnositcaly and then just do flow control in Wyoming.

HarvsG · 2023-10-22T14:40:28Z

Possibly fixed inhttps://github.com/home-assistant/addons/pull/3260, untested

ThreepE0 · 2023-10-22T15:20:12Z

I’m just pointing out that your question was answered pretty clearly: it’s not supported yet. I’d guess this is more on the listener side, as (if I understand correctly,) the audio needs to come in and be forked to multiple pipelines.

to be honest Im a bit confused as well: if the endpoint is just streaming audio, then how is the second word detected at all? Ill have to dig in to things later today.

I got something similar working with Porcupine streaming to a server and handing off to HA, then found Openwakeword was released so Im starting over. Not sure if I should just write a streamer esp firmware and corresponding listener that sends to multiple pipelines, or if I should wait for this to be implemented.

ThreepE0 · 2023-10-22T17:21:34Z

In case it helps, here's some info I was able to track down using some help from ChatGPT this morning:

Component location:

https://github.com/esphome/esphome/tree/dev/esphome/components/voice_assistant

How the ESP Device Knows to Stream to the Home Assistant URL:

The ESP device uses a UDP socket to send audio data. The destination address and port are set when the Home Assistant server sends a "start streaming" message to the ESP device.
The function start_streaming(struct sockaddr_storage *addr, uint16_t port) is responsible for setting the destination address and port for the UDP socket. The address and port are provided by the Home Assistant server.

How the ESP Device Knows Which Pipeline It's Streaming To:

The ESP device doesn't seem to be aware of which pipeline it's streaming to. It simply starts or stops streaming based on commands from the Home Assistant server.
The function request_start(bool continuous, bool silence_detection) is used to start streaming. It sets the continuous_ and silence_detection_ flags based on the parameters it receives.
The function request_stop() is used to stop streaming. It sets the continuous_ flag to false.

Other Relevant Information:

The code uses different states to manage the voice assistant's behavior, such as IDLE, START_MICROPHONE, STREAMING_MICROPHONE, etc.
The function loop() contains the main logic for transitioning between these states and handling audio streaming.

Thoughts:

So far it does sound like this needs to be sorted out on the listening end. Not sure if that'd be done in wyoming, ha core, or somewhere else.

What I might start with is running vanilla OpenWakeWord on another virtual machine, and using it to kick Whisper off. I've got most of the code working to do that already from my Picovoice/Porcupine implementation. That'll get me up and running until

I learn a bit more how the code works and/or
The functionality you've requested here is implemented, as I'm looking forward to it to

BTW I love your profile description HarvsG

HarvsG · 2023-12-20T13:08:43Z

Started a feature request:
https://community.home-assistant.io/t/voice-multiple-voice-assistants-1-wake-word-and-pipeline/658620

HarvsG · 2023-12-20T17:18:53Z

Closing issue as this is a feature request which can be tracked here:
https://community.home-assistant.io/t/voice-multiple-voice-assistants-1-wake-word-and-pipeline/658620

home-assistant bot added the integration: wyoming label Oct 13, 2023

home-assistant bot assigned balloob and synesthesiam Oct 13, 2023

HarvsG closed this as completed Dec 20, 2023

github-actions bot locked and limited conversation to collaborators Jan 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wyoming ignores second assistant's wake word. #101942

Wyoming ignores second assistant's wake word. #101942

HarvsG commented Oct 13, 2023 •

edited

Loading

home-assistant bot commented Oct 13, 2023

HarvsG commented Oct 13, 2023 •

edited

Loading

synesthesiam commented Oct 13, 2023

HarvsG commented Oct 15, 2023

ThreepE0 commented Oct 16, 2023

HarvsG commented Oct 16, 2023 •

edited

Loading

HarvsG commented Oct 22, 2023

ThreepE0 commented Oct 22, 2023 •

edited

Loading

ThreepE0 commented Oct 22, 2023 •

edited

Loading

HarvsG commented Dec 20, 2023

HarvsG commented Dec 20, 2023

Wyoming ignores second assistant's wake word. #101942

Wyoming ignores second assistant's wake word. #101942

Comments

HarvsG commented Oct 13, 2023 • edited Loading

The problem

What version of Home Assistant Core has the issue?

What was the last working version of Home Assistant Core?

What type of installation are you running?

Integration causing the issue

Link to integration documentation on our website

Diagnostics information

Example YAML snippet

Anything in the logs that might be useful for us?

Additional information

home-assistant bot commented Oct 13, 2023

HarvsG commented Oct 13, 2023 • edited Loading

synesthesiam commented Oct 13, 2023

HarvsG commented Oct 15, 2023

ThreepE0 commented Oct 16, 2023

HarvsG commented Oct 16, 2023 • edited Loading

HarvsG commented Oct 22, 2023

ThreepE0 commented Oct 22, 2023 • edited Loading

ThreepE0 commented Oct 22, 2023 • edited Loading

Component location:

How the ESP Device Knows to Stream to the Home Assistant URL:

How the ESP Device Knows Which Pipeline It's Streaming To:

Other Relevant Information:

Thoughts:

HarvsG commented Dec 20, 2023

HarvsG commented Dec 20, 2023

HarvsG commented Oct 13, 2023 •

edited

Loading

HarvsG commented Oct 13, 2023 •

edited

Loading

HarvsG commented Oct 16, 2023 •

edited

Loading

ThreepE0 commented Oct 22, 2023 •

edited

Loading

ThreepE0 commented Oct 22, 2023 •

edited

Loading