Cannot connect to ZMK keyboard over Bluetooth on Pop_OS! 22.04 + Intel AX200 controller

[sethforprivacy]

Hi all,

For some reason (I believe after updating from Pop_OS! 20.10 to 22.04), I am no longer able to connect to either of my ZMK firmware keyboards over Bluetooth. I've collected all the logs I can think of below:

Bluetoothctl output:
Failed to pair: org.bluez.Error.AuthenticationFailed

dmesg output:
[ 9738.118236] Bluetooth: hci0: unexpected SMP command 0x0b from e1:ed:df:d8:a0:25
[ 9738.232020] Bluetooth: hci0: Opcode 0x 401 failed: -16

systemctl journal output:
src/device.c:load_gatt_db() No cache for E1:ED:DF:D8:A0:25

USB logging output:
[00:03:08.153,594] <dbg> zmk: zmk_endpoints_send_report: usage page 0x07
[00:03:19.176,818] <dbg> zmk: connected: Connected thread: 0x20004d90
[00:03:19.176,910] <dbg> zmk: connected: Connected 44:AF:28:56:A6:C0 (public)
[00:03:19.178,405] <dbg> zmk: update_advertising: advertising from 0 to 2
[00:03:19.213,256] <inf> bas: BAS Notifications enabled
[00:03:19.300,628] <err> zmk: Security failed: 44:AF:28:56:A6:C0 (public) level 1 err 4
[00:03:19.390,991] <inf> bas: BAS Notifications disabled
[00:03:19.393,676] <dbg> zmk: disconnected: Disconnected from 44:AF:28:56:A6:C0 (public) (reason 0x05)
[00:03:19.393,707] <dbg> zmk: update_advertising: advertising from 2 to 2

The Bluetooth controller is the built-in controller in the Intel AX200 chipset. I've tested all things I can think of, including manually installing the latest linux-firmware, reinstalling Pop_OS! from scratch, etc.

The connection does work properly on Windows and on another Linux laptop, plus my Android phone.

https://paste.sethforprivacy.com/?d908b857102a6e0d#6Csv3EbvFSz7HeSGTNCzdJupZkze4ZxAmoqVs3fVa5Wr

[sethforprivacy]

A friend of mine more familiar with ZMK suggested clearing all Bluetooth profiles on the keyboard (not just the one I was trying to use) and after doing so the issue is resolved.

If I can provide any additional logs to help track down this bug, please let me know -- I will leave this issue open for a bit in case that would be helpful!

[Huckies]

Maybe the system upgrade wiped all the bonding informations, so the device can not be recognized anymore, and a sweep of Bluetooth profiles on keyboard may remove all the redundant bonds so they can be re-paired and re-bonded again.

[sethforprivacy]

Maybe the system upgrade wiped all the bonding informations, so the device can not be recognized anymore, and a sweep of Bluetooth profiles on keyboard may remove all the redundant bonds so they can be re-paired and re-bonded again.

The problem here is that the issue existed on two different ZMK-based keyboards, one of which was never connected before the upgrade to 22.04 as it didn't exist then.

I also cleared the first 4 profiles (the only ones I have ever tried to use on this keyboard) and that didn't resolve the issue, it was only resolved after clearing all profiles. Seems to be a bug of some sort that requires clearing all profiles from storage before you can properly connect in some scenarios.

[19216801-Franz]

I have the same recurring issue, running ZMK on two nice!nanos for my new split keyboard. Everytime I clear the profiles / reflash the firmware the issue goes away, but returns after one successful connected session using the keyboard over Bluetooth. Any Updates on this by anybody?

[Runeword]

I have the same issue on nixos 23.11 Failed to pair: org.bluez.Error.AuthenticationFailed

I have done all the following : Removing the device, clearing all bluetooth profiles and flashing a settings reset firmware.
After several attempts, I managed to pair the device in sudo mode.

But later I had to connect the device via usb to charge the battery, and after that the same error occur again when I try to pair the device.

[Runeword]

Finally, pairing with bluetuith instead of bluetoothctl resolved the issue in my case.
https://github.com/darkhz/bluetuith

[neyb]

Same issue on a glove80. Definitely a zmk strange behaviour I cannot understand (clear all only worked for me)

[AlexandreBossard]

Same issue, when upgraded to kernel 6.6.2. I tried clearing all profiles and reflashing, all I get is a connect and an immediate disconnect:

kernel: Bluetooth: hci0: unexpected SMP command 0x0b from eb:28:2c:3b:2e:ce
bluedevil-wizard[8227]: kf.bluezqt: PendingCall Error: "Authentication Canceled"

Edit: so the kernel is fine, the issue is related to the arch package linux-firmware-20231110.74158e7a-1. I have no idea who is to blame here.

[neyb]

happens to me again... this time clearing all profiles did not help... (kernel 6.6.3)

[Huckies]

From the Bluetooth protocol point of view:
Authentication failure is most likely to happen with a set of missing stored keys on central side, or to put it more plainly the host side (e.g. smartphones, PCs...). If the key is also/only missing on the peripheral side, central will initiate another pairing procedure to get a new set of keys so all the problems will gone.

From the keyboard view:
There must be a upper limit for the count of stored keys, when the limit is breached, keys can get overlapped, discarded... and a similar restriction can consist in each slot, so a clearing of all profiles is always promising.

[MangoIV]

I first thought this was a kernel issue (works fine on linux 6.1, which I pinned for the time being) but I'm now also using a low energy pointing device which works perfectly fine, I'm starting to think that it's at least partially the fault of ZMK that this does not work

[MangoIV]

• chip: nrf52840
• kernel: 6.6.2 zen, has not been working since 6.4

[MangoIV]

this is pretty serious, if ZMK doesn't support all linux kernels >6.4

[petejohanson]

this is pretty serious, if ZMK doesn't support all linux kernels >6.4

Can you get a full btmon dump during a failed connection, so we can see what's afoot?

[MangoIV]

I will try to do so @petejohanson do you know someone who is running a zmk keyboard on a newer kernel where it's working fine? I have had this issue on all my nrf52840 boards for a while now... I have also reported an issue in the nixos repository, some other folks also reported problems with BLE on newer kernel and wanted to take it to the kernel bug report, the issue is that now I'm using another device that is BLE and works just fine so I started thinking it's not purely the kernels fault.

[petejohanson]

I will try to do so @petejohanson do you know someone who is running a zmk keyboard on a newer kernel where it's working fine? I have had this issue on all my nrf52840 boards for a while now... I have also reported an issue in the nixos repository, some other folks also reported problems with BLE on newer kernel and wanted to take it to the kernel bug report, the issue is that now I'm using another device that is BLE and works just fine so I started thinking it's not purely the kernels fault.

I will check my recently updated Fedora laptop to see the specific kernel version, but otherwise I've not been tracking anything at that granularity at this point. I will need to do so if we see a trend.

Can you also try my Zephyr 3.5 PR to see if anything improves?

[MangoIV]

I have not updated zmk for a while, I will try this first. If issues persist, I will put logs here anyway.

This is first running connnect, then after this fails pair.

le-chiff-ble-connect-6-6-zen.log

[MangoIV]

issues persist on firmware built with the latest 3.5 container

[petejohanson]

issues persist on firmware built with the latest 3.5 container

Just to be clear, is it also built using the 3.5 Zephyr branch of ZMK?

[MangoIV]

https://hub.docker.com/layers/zmkfirmware/zmk-build-arm/3.5-branch-20231114235846-3.5.0-0.16.3-6c450465d1dd-6870957618/images/sha256-33cffa362f3613197c0c37788415271f534b80f168bb56650ba1419d2f03d46d?context=explore

I used this one...

[petejohanson]

https://hub.docker.com/layers/zmkfirmware/zmk-build-arm/3.5-branch-20231114235846-3.5.0-0.16.3-6c450465d1dd-6870957618/images/sha256-33cffa362f3613197c0c37788415271f534b80f168bb56650ba1419d2f03d46d?context=explore

I used this one...

You'd need to actually use the ZMK branch from #1995 and west update. Just using the new container won't use a new Zephyr version under the hood.

[MangoIV]

my github workflow runs west update; does the container not have the 3.5 zephyr? why was it named that?

[petejohanson]

my github workflow runs west update; does the container not have the 3.5 zephyr? why was it named that?

It includes all the SDK and python dependencies needed to build something targeting Zephyr 3.5.

[MangoIV]

NixOS/nixpkgs#242313 (comment)

[petejohanson]

I have not updated zmk for a while, I will try this first. If issues persist, I will put logs here anyway.

This is first running connnect, then after this fails pair.

le-chiff-ble-connect-6-6-zen.log

If this is logs from btmon then it's corrupted, perhaps.... Can we get btmon and dmesg output during the failure to connect?

[MangoIV]

btmon --read works fine for me, apparently it uses some binary format when doing --write

[petejohanson]

Another user reporting issues had a dmesg output indicating it was unhappy with the SMP security request packet... I do wonder if maybe using #2046 and enabling CONFIG_BT_GATT_AUTO_SEC_REQ=y will help.

[petejohanson]

btmon --read works fine for me, apparently it uses some binary format when doing --write

Ah, didn't know it had some binary format support. Looking now.

[petejohanson]

I also notice that the disconnects in both cases occur after the upgrade to 2M PHY... Perhaps related?

> HCI Event: LE Meta Event (0x3e) plen 6                                                                                                  #45 [hci0] 7.254650
      LE PHY Update Complete (0x0c)
        Status: Success (0x00)
        Handle: 16 Address: D7:4E:99:F1:BE:98 (Static)
        TX PHY: LE 2M (0x02)
        RX PHY: LE 2M (0x02)

[vkopio]

Also confirming that CONFIG_BT_CTLR_PHY_2M=n fixed the connectivity issues between the left half of my board (nice!nano v2) and my laptop, but so far the right half cannot connect to the left half. There is a brief moment from time to time when the connection is successful (according to the right screen) but the connection drops immediately.

Flashing the reset firmware did not help and neither did the other proposed options.

[caksoylar]

There is now an aggregate setting called CONFIG_ZMK_BLE_EXPERIMENTAL_CONN which you can set to have a combination of the settings I shared earlier applied. Here is a copy of the call for testers that @petejohanson sent out on Discord:

I could use some help with testing some core changes to our BLE connection settings that are aimed to improve our reliability across all host OS/versions. You don't even need to use a different branch (assuming you are on main). For details on what the config enables, see https://zmk.dev/docs/config/bluetooth#kconfig

To test:

• Enable CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y in your config, rebuild and flash to your keeb. If using a split, please flash both sides.
• You may need to re-pair to hosts, but maybe not. If you do need to, please make a note.
• Test for a day or two.
• Report your testing experience/results in the newly created thread started with this message.

Thanks in advance!

[gyordanov]

@caksoylar thank you !

CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y seems to fix my ble issues and the two halves connect right away and seem to stay connected !

[vkopio]

Looks like the CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y works for me as well. Only had to flash both halves and they started working immediately.

[slabua]

I'm on Fedora 39 and CONFIG_ZMK_BLE_EXPERIMENTAL_CONN did not fix the problem.

No problem whatsoever on Ubuntu 18 (5.4) while I can't see the keyboard at all on Ubuntu 20 (5.15).

[alistairmagee]

Two computers running Debian Bookworm,

• desktop running KDE Plasma on X11
• surface go tablet running a custom kernel form surface-linux project and Gnome Wayland

Suddenly had connection issues with the tablet when the kernel upgraded to 6.6.11. Wouldn't connect to bluetooth, secondary side would not connect to primary when connected via usb. No issues with Desktop computer (or connecting to a Mac)

using CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y fixed it for me.

[slabua]

To those who have solved it with CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y,
do you also have CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y?

[xopclabs]

I'm having this issue on NixOS (kernel 6.1.72), yesterday everything was working fine, today the ability to connect seems random. I've tried flashing with CONFIG_ZMK_BLE_EXPERIMENTA_CONN=y; flashing a reset firmware in prior; checkout zephyr 3.5, west update and build from there. No luck!

[gyordanov]

i did not need CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y
and CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y is working fine with 6.7.0 and linux-firmware from few days ago (arch linux)

[dilshod]

1. CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y helps fixing connecting to MacBook Pro 2017. Without this option it never connects.
2. With this option It is loosing connection to the second half split after turning off and on (is it changing physical address after restart??), and only flashing settings_reset helps reconnecting.

[slabua]

Still no luck. And it is extremely problematic when the keyboard disconnects while typing as the last key stays stuck repeating itself for n times until it reconnects again.

[alistairmagee]

To those who have solved it with CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y, do you also have CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y?

I only used CONFIG_ZMK_BLE_EXPERIMENTAL_CONN=y as I read through the whole thread before attempting a fix and saw that recommended as a catch all. As it fixed the issue for me I didn't try anything else.

What does CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y do? Would it stop my desktop running KDE plasma from asking me to confirm the connection when I first pair or re-pair my keyboard?

[caksoylar]

What does CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y do? Would it stop my desktop running KDE plasma from asking me to confirm the connection when I first pair or re-pair my keyboard?

You can check it here: https://zmk.dev/docs/config/bluetooth#kconfig

It enables passkey entry requirement when pairing, where it asks you to enter a 6 digit pin and press enter. That sounds like what you are describing.

[alistairmagee]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

[petejohanson]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

You can use both halves of a split to enter the security code.

[alistairmagee]

This just broke again today. I don't know what changed between now and last night, I'm not aware of any software updates that happened on my laptop. I tried adding the CONFIG_ZMK_BLE_EXPERIMENTAL_SEC=y flag but as only one half connects to my computer, I can't input the security code, so I removed it again. Is this a problem with ZMK, the linux kernel, or Zephyr?

You can use both halves of a split to enter the security code.

When this bug occurs, the second half doesn't work. Even while primary half is connected via USB.

[petejohanson]

What is the keyboard hardware? Are you possibly fully draining the battery on one or both sides?

[alistairmagee]

Sofle split that I have made a permanent fixed base for (unsplit). They share power over a TRRS cable (permanently plugged in, no chance of damage from insertion and removal, keyboards can't move relative to each other and no room for cables to move) so when one side is powered both are powered.

…

On Tue, 13 Feb 2024 at 01:29, Pete Johanson ***@***.***> wrote: What is the keyboard hardware? Are you possibly fully draining the battery on one or both sides? — Reply to this email directly, view it on GitHub <#1487 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABPNS32ZK7CRWDJUO3OAUSLYTIYOZAVCNFSM6AAAAAARBPH42KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZYHA4TQOJSGM> . You are receiving this because you commented.Message ID: ***@***.***>

[slabua]

As a note, perhaps the issue title should be changed. I guess a lot of people are having the very same issue with indeed different distros and/or configurations.

[Ay1tsMe]

Hoping there is some sort of fix for this. My keyboard which has two nice nano v2s is able to be paired with my android phone and another Linux laptop that has a AX201 Bluetooth controller. But it isn't even discoverable on my PC that has an AX200 controller. I went and bought another Bluetooth controller that was also an AX200 because I thought my old one was faulty but it still doesn't discover it so there must be some sort of Bluetooth issue with zmk connecting to these controllers. Hoping there is some sort of fix soon.

[lesshonor]

[My keyboard] isn't even discoverable on my PC

Your problem does not match the rest of the messages in this issue, then. All of the posts here involve pairing problems, not discovery problems.

[alistairmagee]

Some more points for troubleshooting.

I just got home from a long trip where I had to buy a cheap Bluetooth keyboard for my tablet as I couldn't get this fixed while on the road. I thought maybe that there was an actual error with my nice!nanos due to the weirdness I was having. I could connect the left hand, primary side to my computer with USB cable or Bluetooth. No communication between the halves would happen at this point. I actually had to change my keymap to add a way to get to my "raise" layer from left side to test things with clearing pairings and stuff.

I tried flashing each half with the other sides firmware (after flashing the reset firmware). This is what made me think it was a hardware issue as I could connect the "new" primary half via USB but not via Bluetooth. So maybe all my issues were with a n!n with broken bluetooth. I tested with my android phone to same results. Could connect to Left hand side when it was primary, couldn't connect to Right hand side. No communications between halves.

Back home on my desktop, connected via USB keyboard worked immediately. Both sides communicating as normal. Re-pairing the Bluetooth, it also appears to work fine until I tried changing Bluetooth profiles. Both profile 1 and 2, BT_SEL_0 and BT_SEL_1 were connected to the computer. Swapping between them changed nothing. Pressing BT_SEL_2 stopped input from the keyboard to the computer.

I tried pairing my tablet and it now paired correctly. I put in on slot 3, however it had the same behaviour. BT_SEL_0 would not stop input to the computer. I could swap computers by selecting slot 2 or 3, but slot 1 would just continue on the last computer connected.

This might actually be a hardware issue, but I'm writing everything at this point as I don't know what is going on.

Edit: Bluetooth profile issues are because that key switch was breaking. Ignore that.

TL;DR

1. Surface Go Tablet running Debian Linux, Bookworm, Gnome, Wayland, Kernel 6.7.2-surface-1 would connect to primary half of ZMK split but halves would not communicate with each other

2. Same issue on Android phone. No windows or Mac devices to test with at the time.

3. ZMK split does connect to Desktop PC running Debian Linux, Bookworm, KDE Plasma, (Edit 2: Actually first connected on wayland so this isn't a wayland bug) Xorg or Wayland, Kernel 6.1.0-13-amd64

4. However Bluetooth profile 1 appears to be working incorrectly.

5. After connecting to Desktop, keyboard connects to tablet again correctly.

Please tell me how I can help further.

My current keymap and conf files, in case I have done something obviously wrong and not noticed. I've enabled logging but I don't really know what I am doing with it.

[vvelarde]

Clearing all bt profiles worked for me. Thanks!

[iBot1986]

Hi,
I am not sure, this problem belongs to here;
I have some strange connection troubles with two different Windows machines and a nice!nano v 2.0 (Nordic nRF52840) custom keyboard. (no split keyboard).
Most of the time I must reconnect the keyboard on the computers (remove keyboard from Bluetooth devices and reconnect).
Sometimes it works like this, but sometimes I even must clear all Bluetooth profiles from the keyboard (BT_CLR_ALL) before I am able to reconnect.
The Keyboard has two multiple Bluetooth profiles available to select (BT_SEL 0-3).

I have tried with different config variations. Actually, I am going with CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y

Do you have any suggestions how to solve or debug this?

[slabua]

Still no luck with Fedora latest.

[sadekbaroudi]

So, thanks to @iBot1986
This issue is finally resolved for me. I haven't been able to use ZMK on two of my machines for years... I enabled the CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES option in my config, and it completely addressed the issue after entering the pairing security code. I realize that you posted it to note that it didn't help you in this case, but that was the issue for me. So I must thank you greatly for noting that in your comment.

This may be the issue for others as well, so I just wanted to also encourage others to try that.

[iBot1986]

@sadekbaroudi : Glad I could help!
I must admit that since I have written that comment, the problem haven't occur any more. And now I have even one additional Linux installation.
Maybe the problem caused by the more frequent flashes of the firmware, or also bad handling from my side...
Anyway, it seems CONFIG_ZMK_BLE_EXPERIMENTAL_FEATURES=y helped also for me.