Un-muting microphone after long mute gives broken talon

[EdJoJob]

I mute the microphone in the microphone hardware, and leave talon and MacOS running. When I unmute I have the below in the log and Talon does not respond to my commands.

   12:       lib/python3.9/threading.py:912|
   11:       lib/python3.9/threading.py:954|
   10:       lib/python3.9/threading.py:892|
    9:                    talon/cron.py:110|
    8: ------------------------------------# cron thread
    7:                    talon/cron.py:75 |
    6:          talon/scripting/rctx.py:201|
    5: ------------------------------------# 'cron' main:call()
    4:                     talon/vad.py:16 |
    3: talon/scripting/speech_system.py:284|
    2:             talon/engines/w2l.py:724|
    1:             talon/engines/w2l.py:309|
2021-01-04 12:46:30 WARNING [watchdog] "talon.vad.<lambda>" @4.0s (stalled)
2021-01-04 12:46:32 WARNING [watchdog] "talon.vad.<lambda>" @6.0s (stalled)
2021-01-04 12:46:34 WARNING [watchdog] "talon.vad.<lambda>" @8.0s (stalled)
2021-01-04 12:46:35 ERROR cb error topic="cron" cb=<function _vad_cb.<locals>.<lambda> at 0x121b8cf70>
   14:       lib/python3.9/threading.py:912|
   13:       lib/python3.9/threading.py:954|
   12:       lib/python3.9/threading.py:892|
   11:                    talon/cron.py:110|
   10: ------------------------------------# cron thread
    9:                    talon/cron.py:75 |
    8:          talon/scripting/rctx.py:203|
    7: ------------------------------------# 'cron' main:<lambda>()
    6: ------------------------------------# stack splice
    5:          talon/scripting/rctx.py:201|
    4:                     talon/vad.py:16 |
    3: talon/scripting/speech_system.py:284|
    2:             talon/engines/w2l.py:724|
    1:             talon/engines/w2l.py:311|
RuntimeError: w2l_engine_forward() returned NULL emission
2021-01-04 12:46:38 WARNING [watchdog] "talon.vad.<lambda>" @2.0s

My OS is MacOS Big Sur (On intel silicon)

At this time I am "fixing" the issue by restarting Talon entirely

[lunixbochs]

What kind of microphone? How are you (un)muting it? Does the posted error happen each time you speak or just once when you unmute? If you edit engines.py you can set debug=True to get more information about utterances (you should edit engines.py long before getting into the bad unmuted state though, as editing the file will reload your engine)

[EdJoJob]

The microphone is a blue yeti. I am muting using the on mic mute button.
The watch dog happens each time I speak once it starts happening.
The NULL message happened each time I muted or unmuted.
It has happened twice today, each time after about 1hr of being muted.

I will I try with debug=True and I'll see if it happens again.

[EdJoJob]

Actually as it turns out, that was with debug=True

[lunixbochs]

Can you record externally with QuickTime player around the time the problem happens?

[EdJoJob]

I will if I can get it to happen again, sure

[EdJoJob]

Talon.NULL.emission.error.mp4

This is what I managed to record. Now that it is happening the NULL emission is happening for every sound that would otherwise be checked (speech, loud keyboard etc) and the stalled lambdas happen interspersed with the NULL's

This time I got the error after 30 mins of silence

[lunixbochs]

Are you using speech.record_all = 1? settings.talon: settings(): speech.record_all = 1 If so can you try replaying a clip while the engine is broken, from the repl? with actions.speech.replay(path)

[EdJoJob]

I am, and the below is the output in the REPL

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "talon/scripting/actions.py", line 56, in __call__
  File "talon/scripting/types.py", line 313, in call
  File "/Applications/Talon.app/Contents/Resources/talon_plugins/record.py", line 105, in replay
    speech_system.engine.engine._on_audio_frame(samples, pad=False)
  File "talon/engines/w2l.py", line 724, in _on_audio_frame
  File "talon/engines/w2l.py", line 311, in forward
RuntimeError: w2l_engine_forward() returned NULL emission

[lunixbochs]

Can you check Console.app for crashes related to w2lxpc?

I recently moved wav2letter on Mac to an XPCService. Null emission means something must be wrong with that service after idling. Maybe you were low on memory and launchd killed the service. I probably need additional logic to handle if the service is killed for some reason.

[EdJoJob]

Yes, there were some crashes in Console.app, which were all at or around a malloc from a quick perusal of the crash.

Example dump at https://gist.github.com/EdJoJob/a9cf95aca6dfc9a65f69a836cfb57b52

[lunixbochs]

Beta 0.1.3-24 has two fixes for this.

1. If w2lxpc crashes, Talon will try to restart it.
2. I added some checks to w2lxpc to maybe prevent a crash.

If (1) happens, there may be a little bit of lag on your first command when coming back, and some warnings in the log. Please let me know if (1) is still happening after you update.

[EdJoJob]

I'll keep an eye out for it. Thanks for the fantastic support.

[EdJoJob]

I got w2l_engine_forward() recovered from NULL in the log, but I needed to go looking for it, because it hasn't interrupted my flow. Personally I am good to call this complete for now.

That being said, there was only the one fail today, compared to many yesterday. I may also have less memory pressure than before

[EdJoJob]

Console dump if it helps is at https://gist.github.com/EdJoJob/3332c9fbe262b2f230b07ad8e8b1bbe5

[lunixbochs]

Fixed in v0.1.4, thanks!