[Bug] RP2040 based keyboard not usable from boot

[george-norton]

I have a kyria rev2 with KB2040 based controllers. When the keyboard is connected to my PC or laptop when they are powered on, the keyboard does not start and the PC does not post until I hit the reset button on the keyboard.

Describe the Bug

I have been able to reproduce the same behaviour by flashing my kyria firmware onto a pi pico board. Using the SWD interface on the pico, I can see that the chip has taken a hard fault and is sitting in the _unhandled_exception vector:

0x100002a2 in _unhandled_exception ()
(gdb) bt
#0  0x100002a2 in _unhandled_exception ()
#1  0x100002a2 in VectorBC ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(gdb) info reg 
r0             0x2000198c	536877452
r1             0x0	0
r2             0xf	15
r3             0x20041f00	537140992
r4             0x2000198c	536877452
r5             0x1000	4096
r6             0x200019d8	536877528
r7             0x0	0
r8             0xffffffff	-1
r9             0xffffffff	-1
r10            0xffffffff	-1
r11            0xffffffff	-1
r12            0x0	0
sp             0x200403b0	0x200403b0
lr             0x100002a3	268436131
pc             0x100002a2	0x100002a2 <_unhandled_exception>
xPSR           0x21000003	553648131
msp            0x200403b0	0x200403b0
psp            0x20040b40	0x20040b40
primask        0x0	0
basepri        0x0	0
faultmask      0x0	0
control        0x0	0

If I unplug and reconnect the board, it starts normally and the debugger shows the keyboard_task running.

System Information

Keyboard: Kyria
Revision (if applicable): 2.1
Operating system: Linux (Ubuntu 18.04)
qmk doctor output:

Ψ QMK Doctor is checking your environment.
Ψ CLI version: 1.1.1
Ψ QMK home: /home/george.norton/src/qmk_firmware
Ψ Detected Linux.
⚠ Missing or outdated udev rules for 'atmel-dfu' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'kiibohd' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'stm32-dfu' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'apm32-dfu' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'gd32v-dfu' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'bootloadhid' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'usbasploader' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'usbtinyisp' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'md-boot' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Detected ModemManager without the necessary udev rules. Please either disable it or set the appropriate udev rules if you are using a Pro Micro.
⚠ Missing or outdated udev rules for 'caterina' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
⚠ Missing or outdated udev rules for 'hid-bootloader' boards. Run 'sudo cp /home/george.norton/src/qmk_firmware/util/udev/50-qmk.rules /etc/udev/rules.d/'.
Ψ Git branch: develop
Ψ Repo version: 0.18.6
⚠ Git has unstashed/uncommitted changes.
⚠ The official repository does not seem to be configured as git remote "upstream".
Ψ CLI installed in virtualenv.
Ψ All dependencies are installed.
Ψ Found arm-none-eabi-gcc version 6.3.1
Ψ Found avr-gcc version 5.4.0
Ψ Found avrdude version 6.3
Ψ Found dfu-util version 0.9
Ψ Found dfu-programmer version 0.6.1
Ψ Submodules are up to date.
Ψ Submodule status:
Ψ - lib/chibios: 2022-06-11 09:09:45 +0000 --  (f836d24b0)
Ψ - lib/chibios-contrib: 2022-07-27 10:46:15 +0200 --  (d03aa9cc)
Ψ - lib/googletest: 2021-06-11 06:37:43 -0700 --  (e2239ee6)
Ψ - lib/lufa: 2022-08-26 12:09:55 +1000 --  (549b97320)
Ψ - lib/pico-sdk: 2022-05-17 19:19:01 +0200 --  (07edde8)
Ψ - lib/printf: 2022-06-29 23:59:58 +0300 --  (c2e3b4e)
Ψ - lib/vusb: 2022-06-13 09:18:17 +1000 --  (819dbc1)
Ψ QMK is ready to go, but minor problems were found

Any keyboard related software installed?

• AutoHotKey (Windows)
• Karabiner (macOS)
• Other:

Additional Context

Issue seen both my Dell OptiPlex 7040 and Lenovo ThinkPad X1 6th gen.

My checkout is at d2ce817.

[KarlK90]

Thank you for the report. From a first test it seems like I can reproduce the issue and will investigate further.

[KarlK90]

@george-norton So I have found the reason why the RP2040 hard faults and opened a PR on the ChibiOS-Contrib repo ChibiOS/ChibiOS-Contrib#339

Please note that this will not fix the problem with your split keyboard being unresponsive (though it will not be frozen) after boot-up due to QMKs USB detection mechanism to determine which side is the main or secondary half. See SPLIT_USB_DETECT under https://docs.qmk.fm/#/feature_split_keyboard?id=hardware-configuration-options. Sometimes enumeration fails with a certain BIOS USB stack or the initial time to reach the point where the BIOS would be able to enumerate your keyboard is longer then SPLIT_USB_TIMEOUT. So one thing you could try is to enhance that timeout and see if this fixes your problem. There are RP2040 controllers like the elite-pi which have a schottky diode that allows VBUS sensing by defining USB_VBUS_PIN but this schottky diode is not found on the KB2040 nor the Sparkfun RP2040 Pro-Micro.

[george-norton]

Thank you for your support. I have cherry picked your fixes and enabled SPLIT_USB_DETECT. The keyboard is still not responsive at boot, but the system does POST more quickly so the behaviour does seem better. I will have a play around with different SPLIT_USB_TIMEOUT values (I have tried 2000 and 2500) when I get a chance.

[KarlK90]

Great 🙂, please also apply these changes to QMK -> #18566 (already merged into develop)

My suggestion would be to start with a fairly high SPLIT_USB_TIMEOUT of 5 seconds and work your way down.

[RagingCactus]

Another option to solve the SPLIT_USB_TIMEOUT issue (in my case one of my machines would need an egregiously long timeout, making the timout not an option for me) is to DIY VBUS detection and using USB_VBUS_PIN instead of the timeout.

I used a comparator and some resistors on a tiny "daughterboard" PCB to detect if 5V is present on the "USB+" pad on the back side of the KB2040 (unfortunately it is not 0V on the slave side, but 3.3V instead). This tiny PCB attaches to the JST connector of the KB2040 to supply 3V3 and GND to the daughterboard and provide a signal if VBUS is present on GP12.

There's probably an easier way to do this if you actually know what you're doing with electronics, but the general idea is to DIY a VBUS detection circuit.

[KarlK90]

Fixed by #18566 on develop and root cause fix with ChibiOS/ChibiOS-Contrib#339 which will be merged to develop soonish

[george-norton]

With a longer timeout my desktop is working well. But I still see strange issues with my laptop, but it's not a big issue for me as I usually use my desktop.

On my laptop, if the keyboard is connected at boot, the OLED screen draws a few blocks in the top left hand corner, then it stops drawing. The reset button doesn't seem to do anything. By this point the laptop is sitting on the bitlocker password prompt. Once windows boots, pressing a key seems to bring the keyboard to life.

[nkromm]

just FYI, I am having the exact same issue with boardsource/lulu/rp2040

[sunaku]

@KarlK90 - Yours is the best clue so far to unraveling this mystery for me: I have a WeAct Studio RP2040 board with Schottky diode that lets me use USB_VBUS_PIN, but this issue still affects me on Lenovo Thinkpad laptops (but strangely not on Dell laptops and not on desktops with ASUS motherboards). Based on your explanation, it seems there is something peculiar about the USB stack in Lenovo hardware that prevents my keyboard from being detected at boot time: it remains undetected even after booting into the OS (both Linux and Windows 11) and I need to physically reconnect it in order to make the OS detect it.

#define USB_VBUS_PIN     GP28
#undef  SPLIT_USB_DETECT

Above is the relevant portion of my keyboard's configuration. Notably, without the undef of SPLIT_USB_DETECT, my Lenovo laptop slows to a crawl at boot time. As soon as I physically disconnect my keyboard's USB connection, the slowness disappears.

I'm using QMK version 0.20.1 (which is newer than #18566) so this issue isn't supposed to happen anymore, but I'm observing the contrary in practice. What can I do to help debug this? I'm writing to you after many failed attempts to solve this via QMK defines.

Thanks for your consideration.

[lesshonor]

this issue still affects me on Lenovo Thinkpad laptops

You'll want to track #19593. Might be helpful if you also indicate which ThinkPad model(s) you're working with. @sunaku

[Schievel1]

FWIW I still have this with an RP2040 and F411 and QMK 0.22.12.
I fixed it on the RP2040 with USB_VBUS_PIN though