BCM4345 (Rpi 3+,Rpi 4 ) - CTRL-EVENT-ASSOC-REJECT on WPA2 AP when using SWSUP and brcmfmac

[ABelliqueux]

EDIT: If you want to skip the noise / TL;DR, see my comment at the bottom. The culprit here is SWSUP (authentication offloading to the firmware). Disabling it fixes things.

Hi,

I've encountered the bug a bug similar to the one described here : https://bugs.launchpad.net/raspbian/+bug/1929746 on a RPI 3A+ and a RPI 4B, using raspios bullseye (2022-09-22-raspios-bullseye-arm64-lite.img.xz). With both available firmware-brcm80211 versions ( 1:20221012-1~bpo11+1+rpt1 and 20210315-3 ), the bug exists.

Bug description

Upon trying to connect to a wifi AP from a RPI 3A+ or RPI 4B (BCM4345), the association fails with CTRL-EVENT-ASSOC-REJECT error messages :

sudo wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant.conf -iwlan0
Successfully initialized wpa_supplicant
wlan0: Trying to associate with **:**:**:**:**:** (SSID='foo' freq=2417 MHz)
wlan0: CTRL-EVENT-ASSOC-REJECT bssid=**:**:**:**:**:** status_code=16
wlan0: Trying to associate with **:**:**:**:**:** (SSID='foo' freq=2417 MHz)
wlan0: CTRL-EVENT-ASSOC-REJECT bssid=**:**:**:**:**:** status_code=16
wlan0: Trying to associate with **:**:**:**:**:** (SSID='foo' freq=2417 MHz)
wlan0: CTRL-EVENT-ASSOC-REJECT bssid=**:**:**:**:**:** status_code=16
[etc]

I tested this with 3 different APs, with various channels and regulatory regions and still obtain the same result. No sign of any request in the AP logs.

Swapping the SD card to a RPI 3 (BCM4343), the connection is working flawlessly.

Workaround

As per the original bug report's author suggestion, disabling SAE and/or SWSUP solves the issue :

There seem to be an incompatibility in the Wifi firmware of the brcmfmac Wifi driver on Raspberry Pi 4 which silently prevents it from associating with certain wireless access points that have 802.11r and/or WPA3 enabled.
[...]
I was able to resolve the issue by loading the brcmfmac driver with the following parameters :

rmmod brcmfmac
modprobe brcmfmac roamoff=1 feature_disable=0x82000

This disables the SAE (WPA3 related) and SWSUP (authentication offloading into the firmware), and fast roaming.

I can confirm that after setting those drivers options in /etc/modprobe.d/, the issue disappeared on both 3A+ and 4B. Furthermore, it did not prevent my Rpi 3B (sans +) to connect normally.

I did try to use each setting on it's own but only the combination of those solved the issue.

Questions

I have yet to find any documentation as to the 0x82000 part and what exact features it points to ( Does it disable both SAE and SWSUP at once ? ) Maybe only SWSUP is the issue here ? I did not find documentation regarding SWSUP, no idea how the original author deduced it was part of the culprit. If anyone has knowledge to share on this, it would be nice !

See this page for informations on that 0x82000 value : https://iwd.wiki.kernel.org/offloading

This will disable both SAE (0x80000) and FWSUP (0x02000). You could disable just one, but this is not suggested due to a bug which prevents EAPoL frames from being forwarded after offloading is used for the first time.

Note : I'm aware of #9 but it seems that it was resolved by upgrading to wpasupplicant 2.9 and I'm using 2:2.9.0-21.

System infos

As well as the original bug's author, I'm using dtoverlay=disable-bt in 'config.txt'.

Kernel:

Linux pi 5.15.76-v7+ #1597 SMP Fri Nov 4 12:13:17 GMT 2022 armv7l GNU/Linux

Broadcom fw :

Firmware: BCM4345/6 wl0: Nov  1 2021 00:37:25 version 7.45.241 (1a2f2fa CY) FWID 01-703fd60

Rpi fw:

Oct 26 2022 11:10:35 
Copyright (c) 2012 Broadcom
version c72ad6b26ff40c91ef776b847436094ee63fabee (clean) (release) (start)

wpa_cli version:

wpa_cli v2.10
Copyright (c) 2004-2022, Jouni Malinen <j@w1.fi> and contributors

dhcpcd version:

dhcpcd 8.1.2
Copyright (c) 2006-2019 Roy Marples
Compiled in features: INET ARP ARPing IPv4LL INET6 DHCPv6 AUTH

Wpa_supplicant version:

 2:2.9.0-21

firmware-brcm80211 version :

1:20221012-1~bpo11+1+rpt1

[pelwell]

Is this not just an aspect of raspberrypi/linux#3619, namely that our kernel trees don't currently support WPA3?

[ABelliqueux]

After looking at the issue you mentionned, I can't be sure it's related, and I suspect it is not because the APs I tried to connect to where using neither WPA3 or 80211w/MFP to my knowledge.

AP side

The APs where configured via NetworkManager's CLI interface nmcli with the following settings :
'ieee80211w/MFP' is disabled by default in NetworkManager.

nmcli con add type wifi ifname wlo1 con-name $SSID autoconnect yes ssid $SSID
nmcli con modify $SSID 802-11-wireless.mode ap 802-11-wireless.band bg ipv4.method shared
nmcli con modify $SSID wifi-sec.key-mgmt wpa-psk # would have value 'sae' for WPA3
nmcli con modify $SSID 802-11-wireless-security.proto rsn
nmcli con modify $SSID 802-11-wireless-security.pairwise ccmp
nmcli con modify $SSID wifi-sec.psk $PASSWD
# Would have to use 
# nmcli con modify $SSID 802-11-wireless-security.pmf 1
# to enable 'ieee80211w/MFP'
nmcli radio wifi on
nmcli con up $SSID

(I also tried without setting RSN/CCMP)

Networkmanager's logs show no request coming from the RPI, arp shows nothing related to it either.

Hardware wise, I tested setting an AP with a Intel Wireless 7265, a Realtek RT8821CU and an Intel Wireless-n 2200, all of which gave the same results.

brcmfmac debug=2

On the client side, running 'brcmfmac' with debug=2 , I noticed some messages that do no exist when using the work around :

In 'dmesg_debug2_notworking.log', circa l.1128 :

[snip]
[   33.387875] brcmfmac: brcmf_notify_vif_event Enter: action 2 flags 1 ifidx 0 bsscfgidx 1
[   33.387933] brcmfmac: brcmf_remove_interface Enter, bsscfgidx=1, ifidx=0
[   33.387952] brcmfmac: brcmf_del_if Enter, bsscfgidx=1, ifidx=0
[snip]
[   34.030816] brcmfmac: brcmf_net_setcarrier Enter, bsscfgidx=0 carrier=0
[   34.030848] brcmfmac: brcmf_txflowblock_if enter: bsscfgidx=0 stop=0x0 reason=4 state=1
[snip]

The logs are attached below.

wpa_supplicant

In the RPI's 'wpa_supplicant.conf' :

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=FR

network={
    ssid="my_ssid"
    psk="foobarfoobar"
}

Running wpa_supplicant with '-dd', I noticed that when the 'roamoff' and 'feature_disable' options are used, connections happens after a 'update channel list' operation :

[snip]
P2P: Update channel list
P2P: channels: 81:1,2,3,4,5,6,7,8,9,10,11,12,13 115:36,40,44,48 116:36,44 117:40,48 124:149,153,157,161 125:149,153,157,161,165 126:149,157 127:153,161 128:36,40,44,48 130:36,40,44,48
P2P: cli_channels:
EAPOL: External notification - portValid=1
EAPOL: External notification - EAP success=1
EAPOL: SUPP_PAE entering state AUTHENTICATING
EAPOL: SUPP_BE entering state SUCCESS
EAP: EAP entering state DISABLED
EAPOL: SUPP_PAE entering state AUTHENTICATED
EAPOL: Supplicant port status: Authorized
nl80211: Set supplicant port authorized for 00:e1:8c:**:**:**
EAPOL: SUPP_BE entering state IDLE
EAPOL authentication completed - result=SUCCESS
nl80211: Set rekey offload
[snip]

where as it seems to loop back to an earlier initialization stage when not using the options :

[snip]
P2P: Update channel list
P2P: channels: 81:1,2,3,4,5,6,7,8,9,10,11,12,13 115:36,40,44,48 116:36,44 117:40,48 128:36,40,44,48 130:36,40,44,48
P2P: cli_channels:
nl80211: Set mode ifindex 2 iftype 2 (STATION)
nl80211: Unsubscribe mgmt frames handle 0x88de0d31 (mode change)
nl80211: Subscribe to mgmt frames with non-AP handle 0x5681d0
nl80211: Register frame type=0xd0 (WLAN_FC_STYPE_ACTION) nl_handle=0x5681d0 match=0104
nl80211: Register frame type=0xd0 (WLAN_FC_STYPE_ACTION) nl_handle=0x5681d0 match=040a
nl80211: Register frame type=0xd0 (WLAN_FC_STYPE_ACTION) nl_handle=0x5681d0 match=040b
nl80211: Register frame type=0xd0 (WLAN_FC_STYPE_ACTION) nl_handle=0x5681d0 match=040c
[snip]

Log files :

DMESG with brcmfmac option debug=2 ( not connecting to AP )

dmesg_debug2_notworking.log

DMESG with brcmfmac option debug=2 roamoff=1 feature_disable=0x82000 ( connecting to AP )

dmesg_debug2_roamoff_feat_disable.log

wpa_supplicant log with brcmfmac option debug=2 ( not connecting to AP )

wpa_dd_notworking.log

wpa_supplicant log with brcmfmac option debug=2 roamoff=1 feature_disable=0x82000 ( connecting to AP )

wpa_dd_roamoff_feat_disable.log

[antonsviridenko]

Is this not just an aspect of raspberrypi/linux#3619, namely that our kernel trees don't currently support WPA3?

@pelwell Do I understand correctly that WPA3 support is not available in Pi version of Linux kernel https://github.com/raspberrypi/linux and even using external USB wi-fi module with some other than 'brcmfmac' driver won't help?

[pelwell]

No, just the brcmfmac used by onboard WiFi interfaces, which is what this issue is about.

[ABelliqueux]

@pelwell : If this was an issue with WPA3 support not being available in the kernel, which resulted in not being able to connect to an AP, even when that AP is not using/enabling WPA3, wouldn't that issue hit a lot more people, i.e ; every raspios user ?

[pelwell]

not being able to connect to an AP, even when that AP is not using/enabling WPA3

That doesn't make a lot of sense. I think what the original launchpad(*) bug reporter was saying is that if the AP has WPA3/SAE enabled then there are connection problems, regardless of whether or not wpa_supplicant is configured to use it. That's not something I've experienced or even heard of before.

(*) That's not a recognised place to report problems of any kind with Raspberry Pi devices.

[ABelliqueux]

I've since found this page that shed a bit of light on the matter :

https://iwd.wiki.kernel.org/offloading

Certain drivers support offloading the 4-way handshake as well as SAE/WPA3 into the firmware. For drivers which do not support user space driven Authenticate/Associate frames this is the only way to enable features such as SAE/WPA3 or Fast Transition roaming.

The brcmfmac driver has a module parameter called 'feature_disable'. It expects a hex value (bitmask) where each bit corresponds to a feature.

Regarding the specific 0x82000 value :

This will disable both SAE (0x80000) and FWSUP (0x02000). You could disable just one, but this is not suggested due to a bug which prevents EAPoL frames from being forwarded after offloading is used for the first time.

Another bug report is mentionned in the linked page :

https://lore.kernel.org/linux-wireless/e1ff2a8eef465eaa76e8d00503d04e0fdd61a878.camel@gmail.com/

Particurlarly :

Expected behavior: Connecting to the 2nd WPA2 network using non-offload
should work.

Actual behavior: Connection/4-way times out due to the firmware not
forwarding any eapol frames to userspace.

In my case, there are no WPA3 enabled AP anywhere near me, so it looks like offloading the 4-way handshake is needed even for WPA2 in this particular case, i.e : the bug is triggered even without using SAE in the first place.

[ABelliqueux]

After fiddling a bit more, I've narrowed down the issue to the FWSUP ( authentication offloading) and confirmed that disabling that only with :

modprobe brcmfmac feature_disable=0x02000 

(that's '0x02000' vs '0x82000' earlier ) solves the issue. I'll edit the title to reflect that.

As mentionned in the post above, using '0x82000' is recommended while the bug triggered by SAE exists, if you want to be on the safe side.

[pelwell]

Thanks for doing some digging on this. As I mentioned above, I've not hit this myself or heard of anyone else doing so until this issue was created.

It's going to take me a while to find the time to fully understand the issues, but it does sound like we should consider adding a module .conf file (in /etc/modprobe.d) to set the feature_disable parameter.

[ABelliqueux]

Great, let me know if I can be of any help with tests/logs, etc.

[pelwell]

OK, I'm more up to speed now. From the iwd wiki page:

Should I disable offload support?
First, if your brcmfmac card is behaving normally after updating to an IWD version which includes offload support you should not turn it off.

I tend to agree. There does seem to be a problem with the 43455 firmware, but so far it has only been reported by a very small number of users. The situation might change if a consumer AP was found to trigger the issue, but for now I feel that a page documenting the workaround is the right way to go. Which is what we have here, thanks to your efforts.