WPA3 could crash/hang the router (WRT3200ACM)

[lss4]

I'm currently trying OpenWrt master on this router and during the process I found that using WPA3 could crash the entire router. It seems to happen when the device leaves the access point (disconnecting or switching to another AP, preferrably on the same device), or when I restart the wireless adapter after changing some settings.

During the process, at first the wireless adapter stops working (all APs disappear), and LuCI starts to become unresponsive. After a few minutes all internet connectivites were cut and LuCI stopped working completely (timeout). At least I can SSH into the router and manually reboot it.

When the problem starts to happen, following messages can be seen in the system log.

[  275.710196] ieee80211 phy1: cmd 0x9122=UpdateEncryption timed out
[  275.716330] ieee80211 phy1: return code: 0x1122
[  275.720879] ieee80211 phy1: timeout: 0x1122
[  275.725087] wlan1: failed to remove key (0, 80:a5:89:c7:e9:53) from hardware (-5)
[  275.732747] ieee80211 phy1: MACREG_REG_INT_CODE: 0x0000
[  295.734426] ieee80211 phy1: cmd 0x9111=SetNewStation timed out
[  295.740288] ieee80211 phy1: return code: 0x1111
[  295.744848] ieee80211 phy1: timeout: 0x1111
[  295.752663] ieee80211 phy1: MACREG_REG_INT_CODE: 0x0000
[  315.745685] ieee80211 phy1: cmd 0x801d=MEMAddrAccess timed out
[  315.751554] ieee80211 phy1: return code: 0x001d
[  315.756103] ieee80211 phy1: timeout: 0x001d
[  315.760317] ieee80211 phy1: MACREG_REG_INT_CODE: 0x0000

The router is mostly unresponsive and these log entries are being generated very slowly. It appears as if everything regarding wireless is timing out and is blocking the router, causing certain functionalities to slow down or stop responding.

Switching all APs back to WPA2 and the issues went away. I did not know mwlwifi had issues with WPA3 until recently when I was writing about issues I encountered with this router on OpenWrt master during build and installation processes... so for now, it seems WPA3 is not a good idea...

[johnnyxwan]

This is due to 802.11w (PMF / Protected Management Frames) is required in WPA3. In WPA2, it is optional, and if you enable that in WPA2, it would have the same issue.

The 802.11w issue is quite well known, and currently unsolvable without the wireless adapter firmware being open sourced. I would like to see that one day. Hopefully we can do something, so that Linksys / NXP would like to open source that.

[lss4]

From what are being discussed in this PR... it seems this is not the only issue with these routers' wireless...

Sadly, I don't think getting wireless firmware open-sourced is possible in the near future. From what I saw, wireless is still a very closed ecosystem with all kinds of proprietary technologies and very limited open source contributions from the vendors (and thus ndiswrapper existed to allow some Windows wireless drivers to work on other platforms to a limited extent).

I got the router a few years ago, because at that time it was believed to be the best router ever for the purpose of OpenWrt, with sufficient wireless support. Guess that turned out to be (not entirely) wrong...

Currently I can still use the router without major issues, including 2.4GHz and 5GHz. It's just that I wouldn't dare touching the configurations ever again because every time I touch it I risk getting all APs disabled and it's also very difficult to bring them back up if that happened. From my experience, re-enabling the adapters is not just a simple start/enable after they become disabled.

[Qrbaker]

Just popping by to note that this is still an issue in the 21.02 RC software. This is a bigger issue now as WPA3 is starting to appear in LuCI defaults, and I had a bit of a headscrater resolving this. Changing wireless security to WPA2 only resolves the issue.

[BrainSlayer]

Just popping by to note that this is still an issue in the 21.02 RC software. This is a bigger issue now as WPA3 is starting to appear in LuCI defaults, and I had a bit of a headscrater resolving this. Changing wireless security to WPA2 only resolves the issue.

since i dont have this issue in dd-wrt with wpa3, but willing to help here. can you provide crash logs? (kernel)
one thing need to be considered. openwrt has as far as i know stripped out some softcrypt algorithms from mac80211. this is a big problem for some chipsets and leads to non working wpa3 support. the aes_gmac support in mac80211 is not compiled in in openwrt and not supported by openwrt. this could be the cause.

[Qrbaker]

@BrainSlayer sadly, I rolled back to 19.x firmware becuase I was still having hanging issues. This was at 4AM local time and I threw in the towel. If its useful, I could try and purposefully induce the issue this weekend and pull kernel logs.

Edit: and for the record, if you are talking about /sys/kernel/debug/crashlog, there wasn't one. Ethernet internet remained working; its just DNS, the LuCI web UI, and wireless would all go down. I could visit direct IPs or cached DNS sites over wired connections just fine. I could also still SSH into the router.

I further tried turning on PMF on 19.x and started seeing the hangs, so that is definitely the issue.

[johnnyxwan]

Just popping by to note that this is still an issue in the 21.02 RC software. This is a bigger issue now as WPA3 is starting to appear in LuCI defaults, and I had a bit of a headscrater resolving this. Changing wireless security to WPA2 only resolves the issue.

since i dont have this issue in dd-wrt with wpa3, but willing to help here. can you provide crash logs? (kernel)
one thing need to be considered. openwrt has as far as i know stripped out some softcrypt algorithms from mac80211. this is a big problem for some chipsets and leads to non working wpa3 support. the aes_gmac support in mac80211 is not compiled in in openwrt and not supported by openwrt. this could be the cause.

Interesting to see it working in ddwrt, didn't expect that... I am not sure if the softcrypt algorithms you mentioned are related to the 802.11w (PMF / Protected Management Frames), if it is then this is huge!

Just a hypothesis without any support, see if anybody can verify: I am thinking if there are some problematic crypto operations in the closed-source firmware causing the unresponsive behaviours. The mentioned softcrypt algorithms in mac80211 could be a walkaround for such issues and make it work. Well, just a hypothesis, hopefully someone has more rigid info.

[BrainSlayer]

Just popping by to note that this is still an issue in the 21.02 RC software. This is a bigger issue now as WPA3 is starting to appear in LuCI defaults, and I had a bit of a headscrater resolving this. Changing wireless security to WPA2 only resolves the issue.

since i dont have this issue in dd-wrt with wpa3, but willing to help here. can you provide crash logs? (kernel)
one thing need to be considered. openwrt has as far as i know stripped out some softcrypt algorithms from mac80211. this is a big problem for some chipsets and leads to non working wpa3 support. the aes_gmac support in mac80211 is not compiled in in openwrt and not supported by openwrt. this could be the cause.

Interesting to see it working in ddwrt, didn't expect that... I am not sure if the softcrypt algorithms you mentioned are related to the 802.11w (PMF / Protected Management Frames), if it is then this is huge!

Just a hypothesis without any support, see if anybody can verify: I am thinking if there are some problematic crypto operations in the closed-source firmware causing the unresponsive behaviours. The mentioned softcrypt algorithms in mac80211 could be a walkaround for such issues and make it work. Well, just a hypothesis, hopefully someone has more rigid info.

no. its just the missing gmac implementation in mac80211 in openwrt. in dd-wrt i readded the support for softcrypto gmac. this is required for hostapd / wpa_supplicant authentication only and has nothing todo with the hardware. so you have to mod mac80211 in openwrt. or basicly you have to remove the patch which stripps out the support in mac80211. then you need to add gmac crypto in your kernel config. thats all. and yes its related to PMF too. its a generic bug in openwrt for alot of chipsets.

[dxgldotorg]

In this case, can we explain this in the OpenWRT bug tracker, so they can perhaps re-add the necessary configuration? As of now I have only WPA2 on a config from a previous build of OpenWRT that blows up any time I try to edit it.

[BrainSlayer]

i already talked with a openwrt team member to take care about that issue. i told them about this already 2 years ago. you can also fix it by yourself. just delete 100-remove-cryptoapi-dependencies.patch from packages/mac80211/subsys and then take care to include the neccessary cryptoapi modules in your kernel. CRYPTO_CCM, CRYPTO_GCM etc. this should be all.

[johnnyxwan]

i already talked with a openwrt team member to take care about that issue. i told them about this already 2 years ago. you can also fix it by yourself. just delete 100-remove-cryptoapi-dependencies.patch from packages/mac80211/subsys and then take care to include the neccessary cryptoapi modules in your kernel. CRYPTO_CCM, CRYPTO_GCM etc. this should be all.

cool! but anyone can explain why the log is filled with a bunch of timeouts when this problem exists?

[t-m-w]

This could be great news. I haven't tried it because re-compiling is more than I'm up for at the moment, but if it works (can anyone else confirm?), I think adding this to the official OpenWRT bug tracker is worth a try, to keep a public record if nothing else. I couldn't find a reference to this fix in there anywhere — not even many issues about WPA3 or 802.11w at all, and most have responses saying the user's device doesn't actually use the mwlwifi driver, with no further activity.

[rmandrad]

So, I was able to recompile and make use of mixed WPA2/WP3 for a brief period

I had to remove three patches
131-Revert-mac80211-aes-cmac-switch-to-shash-CMAC-driver.patch
132-mac80211-remove-cmac-dependency.patch
100-remove-cryptoapi-dependencies.patch

together with adding the crypto modules

after reboot I was able to connect with WPA3 ... but after awhile started to get a number of messages like
wlan1: failed to remove key (0, 48:a4:72:5d:86:35) from hardware (-5)
ieee80211 phy1: MACREG_REG_INT_CODE: 0x0000
ieee80211 phy1: cmd 0x9111=SetNewStation timed out

quite a lot of wlan0: AP-STA-POSSIBLE-PSK-MISMATCH although the key is correct

reverted back to previous build. So ... I feel there is more to do than simply remove the patches and add the crypto modules

[BrainSlayer]

the timeout message indicates that your wifi chipset crashed, but mwlwifi hasnt been changed for a long time. i also cannot reproduce this on my device. are there any special patches for mwlwifi in your build? if yes, remove them and start from scratch

[ValCher1961]

Hi @BrainSlayer!
We need to add to hostapd.conf only these keys?
wpa_key_mgmt=SAE
ieee80211w=2

[BrainSlayer]

i'm using

ieee80211w=2
sae_require_mfp=1
sae_password=*******
wpa_key_mgmt=SAE
sae_groups=19 20 21
wpa_pairwise=CCMP
group_mgmt_cipher=AES-128-CMAC
okc=1

wpa_psk instead of sae_password works too, but is more restrictive with key length. its usefull for mixed operation

wpa_pairwise and group_mgmt_cipher depends on the values your card supports. the wrt3200acm is able todo gmp-256 for instance. but not the wrt1900