[BUG] Mod-taps frequently getting stuck

[elkowar]

I've noticed more and more that my home-row modifier mod-tap keys keep getting stuck from time to time. So I did some analysis.

Assuming both S and L are configured as mod-taps for LSHFT and RSHFT respectively, and D and K are both mod-taps for LGUI. A tapping term of 150 is being used.

the following inputs consistently result in the shift modifier being stuck (which is then only resolvable by hitting reset)
HOLD(s), WAIT(200ms), TAP(l), RELEASE(s)

additionally, the following inputs:
HOLD(s), WAIT(200ms), TAP(k), RELEASE(s)
result in no immediate change, until either d or k (either of the mod-taps for LGUI) are tapped again. As soon as either of them is tapped, the gui modifier starts being stuck as held down.

Looking at the logs, I observed the error message ACTIVE_HOLD_TAP_CLEANED_UP_TOO_EARLY

Here is some more log output (this is showing the first input pattern I describe):

logs

[00:00:51.001,953]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:54.885,833]  zmk.kscan_matrix_read: Sending event at 1,2 state on
[00:00:54.885,894]  zmk.zmk_kscan_process_msgq: Row: 1, col: 2, position: 14, pressed: true
[00:00:54.885,925]  zmk.position_state_changed_listener: 14 bubble (no undecided hold_tap active)
[00:00:54.885,955]  zmk.position_state_down: combo: capturing position event 14
[00:00:54.885,955]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:56.227,569]  zmk.kscan_matrix_read: Sending event at 4,3 state on
[00:00:56.227,630]  zmk.zmk_kscan_process_msgq: Row: 4, col: 3, position: 21, pressed: true
[00:00:56.227,661]  zmk.position_state_changed_listener: 21 bubble (no undecided hold_tap active)
[00:00:56.227,691]  zmk.position_state_down: combo: capturing position event 21
[00:00:56.227,722]  zmk.release_pressed_keys: combo: releasing position event 14
[00:00:56.227,722]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 14, binding name: MOD_TAP
[00:00:56.227,752]  zmk.on_hold_tap_binding_pressed: 14 new undecided hold_tap
[00:00:56.227,783]  zmk.release_pressed_keys: combo: reraising position event 21
[00:00:56.227,813]  zmk.decide_hold_tap: 14 decided hold-timer (tap-preferred decision moment timer)
[00:00:56.227,813]  zmk.on_keymap_binding_pressed: position 14 keycode 0x700e1
[00:00:56.227,874]  zmk.hid_listener_keycode_pressed: usage_page 0x07 keycode 0xe1 implicit_mods 0x00 explicit_mods 0x00
[00:00:56.227,874]  zmk.zmk_hid_register_mod: Modifier 1 count 1
[00:00:56.227,874]  zmk.zmk_hid_register_mod: Modifiers set to 0x02
[00:00:56.227,874]  zmk.zmk_hid_implicit_modifiers_press: Modifiers set to 0x02
[00:00:56.227,905]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:56.227,966]  zmk.position_state_changed_listener: 536871936 capturing 21 down event
[00:00:56.227,966]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:56.227,996]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:56.351,837]  zmk.kscan_matrix_read: Sending event at 4,3 state off
[00:00:56.351,898]  zmk.zmk_kscan_process_msgq: Row: 4, col: 3, position: 21, pressed: false
[00:00:56.351,928]  zmk.position_state_changed_listener: 21 bubble (no undecided hold_tap active)
[00:00:56.351,959]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 21, binding name: MOD_TAP
[00:00:56.351,989]  zmk: ACTIVE_HOLD_TAP_CLEANED_UP_TOO_EARLY
[00:00:56.796,051]  zmk.kscan_matrix_read: Sending event at 1,2 state off
[00:00:56.796,142]  zmk.zmk_kscan_process_msgq: Row: 1, col: 2, position: 14, pressed: false
[00:00:56.796,173]  zmk.position_state_changed_listener: 14 bubble (no undecided hold_tap active)
[00:00:56.796,203]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 14, binding name: MOD_TAP
[00:00:56.796,234]  zmk.on_keymap_binding_released: position 14 keycode 0x700e1
[00:00:56.796,264]  zmk.hid_listener_keycode_released: usage_page 0x07 keycode 0xe1 implicit_mods 0x00 explicit_mods 0x00
[00:00:56.796,295]  zmk.zmk_hid_unregister_mod: Modifier 1 count: 0
[00:00:56.796,295]  zmk.zmk_hid_unregister_mod: Modifier 1 released
[00:00:56.796,295]  zmk.zmk_hid_unregister_mod: Modifiers set to 0x00
[00:00:56.796,295]  zmk.zmk_hid_implicit_modifiers_release: Modifiers set to 0x00
[00:00:56.796,325]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:56.796,386]  zmk.on_hold_tap_binding_released: 14 cleaning up hold-tap
[00:00:58.210,021]  zmk.kscan_matrix_read: Sending event at 1,2 state on
[00:00:58.210,083]  zmk.zmk_kscan_process_msgq: Row: 1, col: 2, position: 14, pressed: true
[00:00:58.210,113]  zmk.position_state_changed_listener: 14 bubble (no undecided hold_tap active)
[00:00:58.210,144]  zmk.position_state_down: combo: capturing position event 14
[00:00:58.210,144]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:58.266,754]  zmk.kscan_matrix_read: Sending event at 1,2 state off
[00:00:58.266,876]  zmk.zmk_kscan_process_msgq: Row: 1, col: 2, position: 14, pressed: false
[00:00:58.266,906]  zmk.position_state_changed_listener: 14 bubble (no undecided hold_tap active)
[00:00:58.266,906]  zmk.release_pressed_keys: combo: releasing position event 14
[00:00:58.266,906]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 14, binding name: MOD_TAP
[00:00:58.266,967]  zmk.on_hold_tap_binding_pressed: 14 new undecided hold_tap
[00:00:58.266,998]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 14, binding name: MOD_TAP
[00:00:58.267,059]  zmk.decide_hold_tap: 14 decided tap (tap-preferred decision moment key-up)
[00:00:58.267,059]  zmk.on_keymap_binding_pressed: position 14 keycode 0x70016
[00:00:58.267,089]  zmk.hid_listener_keycode_pressed: usage_page 0x07 keycode 0x16 implicit_mods 0x00 explicit_mods 0x00
[00:00:58.267,120]  zmk.zmk_hid_implicit_modifiers_press: Modifiers set to 0x00
[00:00:58.267,120]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:58.267,181]  zmk.release_captured_events: Releasing key position event for position 21 pressed
[00:00:58.267,181]  zmk.position_state_changed_listener: 21 bubble (no undecided hold_tap active)
[00:00:58.267,211]  zmk.position_state_down: combo: capturing position event 21
[00:00:58.267,211]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:58.267,242]  zmk.on_keymap_binding_released: position 14 keycode 0x70016
[00:00:58.267,272]  zmk.hid_listener_keycode_released: usage_page 0x07 keycode 0x16 implicit_mods 0x00 explicit_mods 0x00
[00:00:58.267,303]  zmk.zmk_hid_implicit_modifiers_release: Modifiers set to 0x00
[00:00:58.267,303]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:58.275,054]  zmk.on_hold_tap_binding_released: 14 cleaning up hold-tap
[00:00:58.275,146]  zmk.release_pressed_keys: combo: releasing position event 21
[00:00:58.275,146]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 21, binding name: MOD_TAP
[00:00:58.275,177]  zmk.on_hold_tap_binding_pressed: 21 new undecided hold_tap
[00:00:58.275,238]  zmk.decide_hold_tap: 21 decided hold-timer (tap-preferred decision moment timer)
[00:00:58.275,268]  zmk.on_keymap_binding_pressed: position 21 keycode 0x700e5
[00:00:58.275,329]  zmk.hid_listener_keycode_pressed: usage_page 0x07 keycode 0xe5 implicit_mods 0x00 explicit_mods 0x00
[00:00:58.275,329]  zmk.zmk_hid_register_mod: Modifier 5 count 1
[00:00:58.275,329]  zmk.zmk_hid_register_mod: Modifiers set to 0x20
[00:00:58.275,329]  zmk.zmk_hid_implicit_modifiers_press: Modifiers set to 0x20
[00:00:58.275,329]  zmk.zmk_endpoints_send_report: usage page 0x07
[00:00:58.603,515]  zmk.kscan_matrix_read: Sending event at 1,1 state on
[00:00:58.603,576]  zmk.zmk_kscan_process_msgq: Row: 1, col: 1, posi

[caksoylar]

Can you share your keymap that you reproduced this with? I tried to reproduce with a test case but haven't had luck so far. I am interested because I also get occasional stuck mods that I haven't been able to log but this sounds like a good lead.

[elkowar]

Sure, here's my config: https://github.com/elkowar/zmk-config/blob/master/config/boards/shields/choctopus44/choctopus44.keymap

@buffet had also reported stuck home row modifiers to me a few days before, but is not encountering them anymore - I assume my case is not the only way to run into this bug

[caksoylar]

Thanks, I don't see GUI mod-taps in that keymap but I guess they would be set up similar to the SHIFT ones.

[elkowar]

Yea, sorry, I removed them as they where causing too many issues currently ^^
The gui mod-taps where set via the still present behavior:

hmgui: homerow_gui {
      compatible = "zmk,behavior-hold-tap";
      label = "HOMEROW_GUI";
      #binding-cells = <2>;
      tapping-term-ms = <200>;
      quick_tap_ms = QUICK_TAP;
      flavor = "tap-preferred";
      bindings = <&kp>, <&kp>;
    };

and where on the d and k keys

[beta-tank]

I also got this bug on my layout: I have configured &mt LSHFT SPACE and &mt RALT BSLH. When I hold Shift and then tap Backslash my Alt gets stuck. I can only fix it by keyboard reboot..

[ad0ll]

This issue occurs about 20-40 times for me a day. My full keymaps are here:

• https://github.com/ad0ll/zmk-config/blob/main/kyria/kyria.keymap
• https://github.com/ad0ll/zmk-config/blob/main/lily58/lily58.keymap

I haven't done the same thorough investigation that elkowar has, but each modifier gets stuck in proportion to how frequently I use it (S->C->M).

(EDIT: Previously I said it usually happens during layer toggles, but that's not accurate. It happens all the time).

[elkowar]

Yea, I still have the issue, and I've mostly had to pretty much just avoid using mod taps completely for now, which is kinda painful,...

[evantravers]

I think I'm encountering this, I'm going to do some extra work to try and reproduce it reliably.

[okke-formsma]

@ad0ll @elkowar what hardware are you using?

[okke-formsma]

does this only happen if you have 'quick-release' and/or 'retro-tap' configured?

[okke-formsma]

The accidental cleanup of the hold-tap must be happening in the timer (the other place where we clean up hold-taps has an additional line of debug prints)

However, this code path can only be followed when hold_tap->work_is_cancelled and that is only set to true after the behavior has been released.

So how can it be possible that the key is not released?

In the keymap examples, retro-tapping is disabled, so that's not the answer either. I don't see how this bug can happen :(

[ad0ll]

@okke-formsma, I'm using Nice!Nano 2.0s and have tested with a Kyria, Corne, and Lily58. I have some unbuilt BlueMicros lying around that I could build, but I'd need a week-ish to get back to you.

My config for home row mods was this before I gave up:

hm: homerow_mods {            
			compatible = "zmk,behavior-hold-tap";            
			label = "HOMEROW_MODS";          
			#binding-cells = <2>;  
			tapping-term-ms = <129>;            
			quick_tap_ms = <400>;            
			flavor = "balanced";        
			bindings = <&kp>, <&kp>;        
		};    

I had tried with retro-tap before, but I'm not seeing it in my config now.

I started hitting this issue about 70-100 times a day, and did a very naive pass at the code. I didn't get the same error messages at elkowar, but used their debugging efforts to make an attempt. The changes I made in this commit (again, I don't know what I'm doing, but thought to experiment) reduced the occurrences down to about 5-8 a day. However, with these changes, when a mod tap issue was triggered, my entire keyboard would cease to work until a reset.

[okke-formsma]

After reading the logs in the initial post more carefully, I think the following sequence may be the root of the issue:

00:00:56.227,966]  zmk.position_state_changed_listener: 536871936 capturing 21 down event
[00:00:56.227,966]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:56.227,996]  zmk.zmk_event_manager_handle_from: Listener captured the event
[00:00:56.351,837]  zmk.kscan_matrix_read: Sending event at 4,3 state off
[00:00:56.351,898]  zmk.zmk_kscan_process_msgq: Row: 4, col: 3, position: 21, pressed: false
[00:00:56.351,928]  zmk.position_state_changed_listener: 21 bubble (no undecided hold_tap active)
[00:00:56.351,959]  zmk.zmk_keymap_apply_position_state: layer: 0 position: 21, binding name: MOD_TAP
[00:00:56.351,989]  zmk: ACTIVE_HOLD_TAP_CLEANED_UP_TOO_EARLY

Told-tap 21 is not cleaned up to early - it was never actually registered because the key down event is still in the queue for hold-tap 14 536871936.

Two seconds later, the key down event for HT 21 is released, causing a key-down without a key-up.

[00:00:58.267,181]  zmk.release_captured_events: Releasing key position event for position 21 pressed

This could be an interplay between combos and hold-taps 🤔

Aaaah the plot thickens - there's a combo at play in the keymap on the positions 14 and 21:

COMBO(twohand_colon,     20, 20, <14 21>,    &kp COLON);

[okke-formsma]

This is weird:

So the position of the hold-tap that's supposedly capturing this keypress is 536871936, which is not a valid position unless you have a large board.

Now, how can the undecided_hold_tap be corrupted like this? 🤔

[evantravers]

That's a very clean number in binary: 536871936 -> 00100000000000000000010000000000

Makes me wonder if there's an overflow, max int size, or similar curmudgeonry occurring.

[mrlinuxfish]

I'm getting this or a similar issue on my hummingbird. The strange thing is I haven't got this issue on any other board using a similar config

[okke-formsma]

@mrlinuxfish do you use different combos on your hummingbird?

[mrlinuxfish]

I have thumb combos (which are also on the Zaphod and and ble-Chiffre). The hummingbird also has overlapping combos on the bottom row to emulate the bottom corner keys

[mrlinuxfish]

I have not noticed any stuck keys after reducing my tapping term to 150 from 200. It may be coincidence or the timing window for this bug is smaller with the shorter tapping term. Either way the tighter tapping term works better for me overall even if it doesn't fix the issue.

Update: it's still having issues on 150 tapping term but the tighter timing seems to mitigate the issue somewhat

[elkowar]

@ad0ll @elkowar what hardware are you using?

I'm using a choctopus44
I don't think I found any options that made a difference although I'm not 100% sure on those. Can't check right now, but I'll try to remember ^^'

Thanks for investigating!

[boekhocg]

Curious, anyone here experiencing this that was not connected to more than one profile? I connected to a second profile and started to constantly get the issue. I disconnected from one and it's been good since. Might be coincidence, but thought I'd ask in case.

[trankillity]

Curious, anyone here experiencing this that was not connected to more than one profile? I connected to a second profile and started to constantly get the issue. I disconnected from one and it's been good since. Might be coincidence, but thought I'd ask in case.

Unfortunately this isn't the case as I only use my Sweep with nice!nano v2s at work. I just tried to switch to Callum-style mods using sticky modifiers on layers on the home row and am experiencing stuck mods almost instantly upon flashing/using them.

The weird thing is that I have a sticky shift on my base layer which has NEVER caused this issue. I also have home row mods on my base layer which have never caused this issue either. Additionally, I have combos (and overlapping combos) against all of the keys used for home row mods and for the mods on the layers.

My full config is here if it helps.

Note: My sticky key mods are using hold-tap behaviours.

[trankillity]

Hey all, have a fair bit more information to add to this issue.

TLDR: This appears to be an issue introduced to the core hold-tap behaviour since 23rd December 2021.

Reasoning:

• I have a thumb button shift key which is controlled by a hold-tap behaviour. The tap of the behaviour is set to &sk LSHIFT while the hold is set to &kp LSHIFT (to prevent sticky from triggering when holding the key and releasing - important for 3D applications)
• Removing the hold-tap behaviour on this key, and changing it directly to &sk LSHIFT prevented stuck shift entirely.
• I have only started to experience this issue after recompiling recently. Prior to that, the last time I updated was 23rd December 2021. Might help to look at any changes made to it since that date.

Hopefully this helps in narrowing down the cause of this bug. It's been pretty annoying in the last few days since recompiling as I need to have a spare keyboard handy to "unstick" the mods.

[halcyonCorsair]

I tried going back to pre-December 23 with no luck.

However I have had some luck by removing some combos:
With this config, I got the issue almost constantly.

However these changes to my combos seem to have made it go away.

[ad0ll]

Chiming in, I cut all of my combos and am still experiencing the issue. As for pre-December, I personally have had this issue consistently since getting nice nanos in Sept 2021

[trankillity]

Small anecdote to add to this issue:

I have just moved to using Callum-style mods instead of mod-taps and the issue has completely disappeared. So definitely appears as if the issue is within the implementation of passing mods to the hold (or more likely, the release) portion of the hold-tap behaviour.

[timwenger]

@elkowar and @okke-formsma.
I often get stuck down home row mods (as in, until I reset, that modifier key is permanently being sent to the PC).
After some debugging, I can confirm that I have the same tag showing ACTIVE_HOLD_TAP_CLEANED_UP_TOO_EARLY (log line 36), and I have the same corrupted key press number, 536871936 (log line 29), as the original poster.

I can reproduce this by typing in Notepad++ with the left hand (main side of the keyboard), which has
- a middle finger on index 15 [&hm LALT S] on Layer 0, and a simple [&kp DOWN] on layer 1.
- a pinky finger on index 25 [&lt L1 Z] to go from layer 0 to layer 1.
- a combo that includes a key at index 25, with a timeout-ms longer than the tapping-term-ms that is set for the corresponding &lt. my combo is: COMBO(num_one, 5000, 13 25, &kp N1), which sends num 1 when keys at index 13 and 25 are pressed within 5000 ms of each other. 5000ms exaggerated for exploiting this ticket.

To reproduce:

1. I press the pinky finger (index 25) to go into layer 1, then press the middle finger (index 15). Altogether, this should press the DOWN key.

2. However, if index 15 is pressed within 5000ms of index 25 being pressed, the DOWN key will not be pressed.

3. Now all keys can be released, it doesn't matter what order they are released in.

4. At this point, if index 15 is tapped, it prints one S, pauses for a quick moment, then prints S's continually until I hit another key, as if I tapped and then held index 15, the S key.

5. Regardless of whether you tapped index 15 in step 4, once any home row mod is tapped, the ALT key (the modifier on index 15) is now stuck (held down) until I reset the keyboard.
   After resetting, I have to hold index 15 (to produce an ALT keystroke) before the computer behaves normally again. Otherwise it behaves as though ALT is still pressed.

Therefore, I can affirm that a combo can affect homerow mod. This behavior was repeatable for other &hm keys, as well as repeatable for keys on the right half.

For now, my work around is to makes sure that my combo timeout-ms is not longer than my &lt tapping-term-ms.
I've attached my keymap file, and a debug log output of the events on steps 1 and 2.

log.txt
corne.keymap.txt

I hope this helps debug this problem!

[urob]

Adding one more observation: This issue got significantly mitigated (EDIT: solved?) for me when using equal timeouts for all my combos.

I used to have overlapping combos with different timeouts on almost all my alpha-keys, and my HRMs would get stuck virtually all the time. Setting all combo timeouts to the same value seemed to have solved the problem (EDIT: as of 11 days later, I haven't yet encountered this issue at all).

Possibly related: #905

[petejohanson]

Please feel free to re-open if you encounter this again w/ latest main. Thanks.