Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenWakeWord has variable behaviour for multiple assistants #3259

Closed
HarvsG opened this issue Oct 13, 2023 · 5 comments
Closed

OpenWakeWord has variable behaviour for multiple assistants #3259

HarvsG opened this issue Oct 13, 2023 · 5 comments
Labels
stale

Comments

@HarvsG
Copy link

HarvsG commented Oct 13, 2023 •
edited
Loading

Describe the issue you are experiencing

The wakeword for non-favourite voice assistants is being ignored

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Which add-on are you reporting an issue with?

OpenWakeWord

What is the version of the add-on?

1.7.0

Steps to reproduce the issue

  1. Configure more than 1 voice assistant, with different wake words (I use a completely nabu-casa pipeline for 'hey rasspy' and a nabu casa/OpenAI ChatGPT for 'hey jarvis'. Set 'hey rasspy' to be your favourite.
  2. Configure a home-assistant satellite
  3. Make plenty of 'hey rasspy' commands and see them succeed, in the OpenWakeWord debug log see single entries of DEBUG:root:Triggered hey_rhasspy_v0.1 (client=8641896508078)
  4. Switch the favourite to being jarvis - see in the debug log that OpenWakeWord loads the 'hey jarvis' model (and does not unload 'hey rhasspy'. See that 'hey jarvis' commands succeeed.
  5. Switch the favourite back to the 'hey rasspy' assistant
  6. Make some 'hey jarvis' commands, from the home-assistant satellite command --debug feed see no acknowledgement of wake-work detection.
  7. In the OpenWakeWork debug log see duplicate entries for wake word detection for the non-favourite voice assistant:
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=8650962437429)
  1. Switch which is the favourite (starred) voice assistant and see the behaviour reverse

System Health information

System Information

version core-2023.10.2
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.11.5
os_name Linux
os_version 6.1.21-v8
arch aarch64
timezone Europe/London
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 5000
Installed Version 1.32.1
Stage running
Available Repositories 1305
Downloaded Repositories 8
Home Assistant Cloud
logged_in true
subscription_expiration November 4, 2023 at 12:00 AM
relayer_connected true
relayer_region eu-central-1
remote_enabled true
remote_connected true
alexa_enabled false
google_enabled true
remote_server eu-central-1-14.ui.nabu.casa
certificate_status ready
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 10.5
update_channel stable
supervisor_version supervisor-2023.10.0
agent_version 1.5.1
docker_version 23.0.6
disk_total 55.7 GB
disk_used 10.5 GB
healthy true
supported true
board rpi4-64
supervisor_api ok
version_api ok
installed_addons RPC Shutdown (2.4), Home Assistant Google Drive Backup (0.111.1), File editor (5.6.0), Tailscale (0.11.1), OpenThread Border Router (2.3.2), ESPHome (2023.9.3), Matter Server (4.9.0), Advanced SSH & Web Terminal (15.0.8), openWakeWord (1.7.0)
Dashboards
dashboards 5
resources 2
views 22
mode storage
Recorder
oldest_recorder_run October 12, 2023 at 1:53 PM
current_recorder_run October 13, 2023 at 12:27 PM
estimated_db_size 351.98 MiB
database_engine sqlite
database_version 3.41.2

Anything in the Supervisor logs that might be useful for us?

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/connection.py:150
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 1:09:12 PM (121 occurrences)
Last logged: 2:51:59 PM

[546757196992] Received binary message for non-existing handler 1
[546928783296] Received binary message for non-existing handler 1
[546739480256] Received binary message for non-existing handler 1
[546805007936] Received binary message for non-existing handler 1
[546805022272] Received binary message for non-existing handler 1

Anything in the add-on logs that might be useful for us?

Logger: homeassistant.components.wyoming.wake_word
Source: components/wyoming/wake_word.py:123
Integration: Wyoming Protocol (documentation, issues)
First occurred: 2:53:34 PM (2 occurrences)
Last logged: 2:53:34 PM

Expected wake word hey_rhasspy_v0.1 but got hey_jarvis_v0.1, skipping

Additional information

pi@redpi:~/homeassistant-satellite $ script/run --host 192.168.0.30 --token eyJredactedTA4Nzg4fQ.CM-43hmuImaukgtrsw2eG6clobjG9kg22mvNOwr94zQ --done-sound sounds/done.wav --vad webrtcvad --debug --mic-device plughw:CARD=Device --debug-recording-dir records --noise-suppression 2 --auto-gain 25 --volume-multiplier 6
DEBUG:__main__:Namespace(host='192.168.0.30', token='eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI0NDA3ZTM4NjNkZmQ0NmY2YTM4MjE0NDJjZmEzMWI4YyIsImlhdCI6MTY5NzE0ODc4OCwiZXhwIjoyMDEyNTA4Nzg4fQ.CM-43hmuImaukgtrsw2eG6clobjG9kg22mvNOwr94zQ', port=8123, api_path='/api', pipeline=None, protocol='http', mic_device='plughw:CARD=Device', snd_device=None, mic_command=None, snd_command=None, snd_command_sample_rate=22050, awake_sound=None, done_sound='sounds/done.wav', volume=1.0, vad='webrtcvad', vad_model=PosixPath('/home/pi/homeassistant-satellite/homeassistant_satellite/models/silero_vad.onnx'), vad_threshold=0.5, vad_trigger_level=3, vad_buffer_chunks=40, wake_buffer_seconds=0, noise_suppression=2, auto_gain=25, volume_multiplier=6.0, udp_mic=None, udp_snd=None, udp_snd_sample_rate=22050, debug_recording_dir='records', debug=True)
DEBUG:__main__:Waiting for speech
DEBUG:__main__:Using webrtc audio processing
DEBUG:root:Microphone command: ['arecord', '-D', 'plughw:CARD=Device', '-r', '16000', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Recording raw data 'stdin' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono
DEBUG:__main__:Speech detected
DEBUG:homeassistant_satellite.remote:{'type': 'auth_required', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'type': 'auth_ok', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'result', 'success': True, 'result': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-start', 'data': {'pipeline': '01gzv00ms6vap6j6fa7cn6t2s4', 'language': 'en', 'runner_data': {'stt_binary_handler_id': 1, 'timeout': 300}}, 'timestamp': '2023-10-13T13:02:40.765878+00:00'}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-start', 'data': {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}, 'timestamp': '2023-10-13T13:02:40.766173+00:00'}}
DEBUG:__main__:wake_word-start {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-end', 'data': {'wake_word_output': {'wake_word_id': 'hey_rhasspy_v0.1', 'timestamp': 1570}}, 'timestamp': '2023-10-13T13:02:42.746383+00:00'}}
DEBUG:__main__:wake_word-end {'wake_word_output': {'wake_word_id': 'hey_rhasspy_v0.1', 'timestamp': 1570}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-start', 'data': {'engine': 'cloud', 'metadata': {'language': 'en-GB', 'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}}, 'timestamp': '2023-10-13T13:02:42.746907+00:00'}}
DEBUG:__main__:stt-start {'engine': 'cloud', 'metadata': {'language': 'en-GB', 'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-vad-start', 'data': {'timestamp': 1890}, 'timestamp': '2023-10-13T13:02:43.269117+00:00'}}
DEBUG:__main__:stt-vad-start {'timestamp': 1890}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-vad-end', 'data': {'timestamp': 2555}, 'timestamp': '2023-10-13T13:02:44.627470+00:00'}}
DEBUG:__main__:stt-vad-end {'timestamp': 2555}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'stt-end', 'data': {'stt_output': {'text': 'Turn off the study lights.'}}, 'timestamp': '2023-10-13T13:02:44.786910+00:00'}}
DEBUG:__main__:stt-end {'stt_output': {'text': 'Turn off the study lights.'}}
DEBUG:root:play ffmpeg: ['ffmpeg', '-i', 'sounds/done.wav', '-f', 'wav', '-ar', '22050', '-ac', '1', '-filter:a', 'volume=1.0', '-']
DEBUG:root:play: ['aplay', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'intent-start', 'data': {'engine': 'homeassistant', 'language': 'en', 'intent_input': 'Turn off the study lights.', 'conversation_id': None, 'device_id': None}, 'timestamp': '2023-10-13T13:02:44.787097+00:00'}}
DEBUG:__main__:intent-start {'engine': 'homeassistant', 'language': 'en', 'intent_input': 'Turn off the study lights.', 'conversation_id': None, 'device_id': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'intent-end', 'data': {'intent_output': {'response': {'speech': {'plain': {'speech': 'Turned off lights', 'extra_data': None}}, 'card': {}, 'language': 'en', 'response_type': 'action_done', 'data': {'targets': [], 'success': [{'name': 'Study', 'type': 'area', 'id': 'study'}, {'name': 'Study Light', 'type': 'entity', 'id': 'light.study_light'}, {'name': 'Study Bulb 4', 'type': 'entity', 'id': 'light.study_bulb_4'}, {'name': 'Study Bulb 3', 'type': 'entity', 'id': 'light.study_bulb_3'}, {'name': 'Study Bulb 2', 'type': 'entity', 'id': 'light.study_bulb_2'}, {'name': 'Study Bulb 1', 'type': 'entity', 'id': 'light.study_bulb_1'}, {'name': 'Study Lamp', 'type': 'entity', 'id': 'light.study_lamp'}], 'failed': []}}, 'conversation_id': None}}, 'timestamp': '2023-10-13T13:02:45.073797+00:00'}}
DEBUG:__main__:intent-end {'intent_output': {'response': {'speech': {'plain': {'speech': 'Turned off lights', 'extra_data': None}}, 'card': {}, 'language': 'en', 'response_type': 'action_done', 'data': {'targets': [], 'success': [{'name': 'Study', 'type': 'area', 'id': 'study'}, {'name': 'Study Light', 'type': 'entity', 'id': 'light.study_light'}, {'name': 'Study Bulb 4', 'type': 'entity', 'id': 'light.study_bulb_4'}, {'name': 'Study Bulb 3', 'type': 'entity', 'id': 'light.study_bulb_3'}, {'name': 'Study Bulb 2', 'type': 'entity', 'id': 'light.study_bulb_2'}, {'name': 'Study Bulb 1', 'type': 'entity', 'id': 'light.study_bulb_1'}, {'name': 'Study Lamp', 'type': 'entity', 'id': 'light.study_lamp'}], 'failed': []}}, 'conversation_id': None}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'tts-start', 'data': {'engine': 'cloud', 'language': 'en-GB', 'voice': 'MaisieNeural', 'tts_input': 'Turned off lights'}, 'timestamp': '2023-10-13T13:02:45.074084+00:00'}}
DEBUG:__main__:tts-start {'engine': 'cloud', 'language': 'en-GB', 'voice': 'MaisieNeural', 'tts_input': 'Turned off lights'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'tts-end', 'data': {'tts_output': {'media_id': 'media-source://tts/cloud?message=Turned+off+lights&language=en-GB&voice=MaisieNeural', 'url': '/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', 'mime_type': 'audio/mpeg'}}, 'timestamp': '2023-10-13T13:02:45.075294+00:00'}}
DEBUG:__main__:tts-end {'tts_output': {'media_id': 'media-source://tts/cloud?message=Turned+off+lights&language=en-GB&voice=MaisieNeural', 'url': '/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', 'mime_type': 'audio/mpeg'}}
DEBUG:root:play ffmpeg: ['ffmpeg', '-i', 'http://192.168.0.30:8123/api/tts_proxy/3ab0fdf873d5e839d27be59ad331f26dc357d73f_en-gb_5230df2865_cloud.audiooutput.mp3', '-f', 'wav', '-ar', '22050', '-ac', '1', '-filter:a', 'volume=1.0', '-']
DEBUG:root:play: ['aplay', '-r', '22050', '-c', '1', '-f', 'S16_LE', '-t', 'raw']
Playing raw data 'stdin' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-end', 'data': None, 'timestamp': '2023-10-13T13:02:45.075435+00:00'}}
DEBUG:__main__:run-end None
DEBUG:homeassistant_satellite.remote:Pipeline finished
DEBUG:__main__:Waiting for speech
DEBUG:__main__:Speech detected
DEBUG:homeassistant_satellite.remote:{'type': 'auth_required', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'type': 'auth_ok', 'ha_version': '2023.10.2'}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'result', 'success': True, 'result': None}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'run-start', 'data': {'pipeline': '01gzv00ms6vap6j6fa7cn6t2s4', 'language': 'en', 'runner_data': {'stt_binary_handler_id': 1, 'timeout': 300}}, 'timestamp': '2023-10-13T13:02:49.766683+00:00'}}
DEBUG:homeassistant_satellite.remote:{'id': 1, 'type': 'event', 'event': {'type': 'wake_word-start', 'data': {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}, 'timestamp': '2023-10-13T13:02:49.767102+00:00'}}
DEBUG:__main__:wake_word-start {'entity_id': 'wake_word.openwakeword', 'metadata': {'format': 'wav', 'codec': 'pcm', 'bit_rate': 16, 'sample_rate': 16000, 'channel': 1}, 'timeout': 3}

DEBUG:root:Triggered hey_rhasspy_v0.1 (client=5826969686261)
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5826969686261
DEBUG:wyoming_openwakeword.handler:Client connected: 5835970430964
DEBUG:wyoming_openwakeword.handler:Receiving audio from client: 5835970430964
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:root:Triggered hey_jarvis_v0.1 (client=5835970430964)
DEBUG:wyoming_openwakeword.handler:Client connected: 5855276917884
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5855276917884
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5855276917884
DEBUG:wyoming_openwakeword.handler:Client connected: 5887450564782
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5887450564782
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5887450564782
DEBUG:wyoming_openwakeword.handler:Client connected: 5919591716748
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5919591716748
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5919591716748
DEBUG:wyoming_openwakeword.handler:Client connected: 5951717174131
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5951717174131
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5951717174131
DEBUG:wyoming_openwakeword.handler:Client connected: 5983853725401
DEBUG:wyoming_openwakeword.handler:Sent info to client: 5983853725401
DEBUG:wyoming_openwakeword.handler:Client disconnected: 5983853725401
DEBUG:wyoming_openwakeword.handler:Client connected: 6016004259331
DEBUG:wyoming_openwakeword.handler:Sent info to client: 6016004259331
DEBUG:wyoming_openwakeword.handler:Client disconnected: 6016004259331
DEBUG:wyoming_openwakeword.handler:Client connected: 6048179180758
DEBUG:wyoming_openwakeword.handler:Sent info to client: 6048179180758
DEBUG:wyoming_openwakeword.handler:Client disconnected: 6048179180758
@HarvsG
Copy link
Author

HarvsG commented Oct 13, 2023

synesthesiam/homeassistant-satellite#13

@HarvsG
Copy link
Author

HarvsG commented Oct 13, 2023

Seeing this error make me think maybe it is a wyoming error

Logger: homeassistant.components.wyoming.wake_word
Source: components/wyoming/wake_word.py:123
Integration: Wyoming Protocol (documentation, issues)
First occurred: 2:53:34 PM (2 occurrences)
Last logged: 2:53:34 PM

Expected wake word hey_rhasspy_v0.1 but got hey_jarvis_v0.1, skipping

@HarvsG HarvsG mentioned this issue Oct 13, 2023
Closed
@HarvsG
Copy link
Author

HarvsG commented Oct 13, 2023 •
edited
Loading

Ok, after some digging, I can refine the behaviour.
If you restart OpenWakeWord it will only detect the wake-word of the currently favourited assistant.
If you switch favourite assistant whilst OpenWakeWord is running, it will load the new wake word. It will still detect the previous favourite (with the duplications in the debug log) but wyoming will skip/ignore the old favourite wake word.

So it seems multiple assistants are not supported in wyoming, and they can be supported in OpenWakeWord via this work-around of favourite-switching

@HarvsG HarvsG changed the title OpenWakeWord does not process second voice asssistant pipeline OpenWakeWord has variable behaviour for multiple assistants Oct 13, 2023
@HarvsG HarvsG mentioned this issue Oct 16, 2023
Closed
@mikey60
Copy link

mikey60 commented Oct 23, 2023 •
edited
Loading

Multiple assistants did not work for me either.
Update: I have two voice assistants working after upgrading openWakeWord Addon to 1.8.1. Of course each voice assistant uses 7-8% of the CPU for the openWakeWord Addon on my Raspberry Pi 4B.

@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Nov 25, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikey60 @HarvsG