mt7981 (Cudy WR3000S): "nl80211: kernel reports: key addition failed" causing multi-minute disconnects during 802.11r FT roaming

[duetre2002]

hostapd/mac80211: "nl80211: kernel reports: key addition failed" during 802.11r FT roaming – reproducible WiFi disconnects, persists across OpenWrt 25.12.4 -> 25.12.5

Summary

On a production 3-AP mesh (3× Cudy WR3000S, MediaTek Filogic/MT7981, originally OpenWrt 25.12.4, now upgraded to 25.12.5) the error nl80211: kernel reports: key addition failed occurs reproducibly whenever 802.11r (Fast Transition) is enabled. In many cases it stays a log-only event (hostapd's internal retry succeeds), but in a significant number of cases it causes real, multi-minute WiFi disconnects on client devices (see attachment 02).

After systematically ruling out every other plausible cause (roaming trigger, PMK derivation, protocol timing parameters, system load from logging), the error was reliably eliminated by fully disabling 802.11r on all radios/routers (see attachment 06). This points to a race condition between hostapd's PTK installation during an FT reassociation and the WLAN_STA_ASSOC state transition in the kernel (net/mac80211/cfg.c, ieee80211_add_key()).

Update: The router has since been upgraded from OpenWrt 25.12.4 to 25.12.5, and FT was re-enabled on all interfaces to retest whether the point release changed anything. The issue reoccurs identically on 25.12.5. The 25.12.5 changelog does not mention any fix touching mac80211/mt76 key installation or FT timing (only a wifi-scripts null-pointer fix for 6GHz-only radios, an EAP station-mode fix, a disabled-virtual-interface-tracking fix, and a hostapd DFS radar notification fix), which is consistent with the issue persisting.

Environment

• Hardware: 3× Cudy WR3000S (MediaTek MT7981 SoC, Filogic target)
• OpenWrt version: originally 25.12.4, now 25.12.5 (see attachment 00 for the full /etc/openwrt_release output - please fill in on the 25.12.5 system before posting)
• WiFi driver: kmod-mt76-core / kmod-mt76-connac, snapshot base 2026.03.19~39c960c3 (as of 25.12.4; please confirm current version on 25.12.5 via attachment 00)
• hostapd/wpad: wpad-mbedtls 2025.08.26~ca266cc2-r2 (as of 25.12.4)
• Network topology: cudy1 is the router for 192.168.179.0/24; cudy2 and cudy3 act as APs, connected via tagged 802.1q VLAN (VLAN ID 179) through unmanaged switches on the same L2 network. All three routers share the same SSID/PSK per band and the same 802.11r mobility domain to allow seamless client roaming between APs.
• Affected radios: predominantly 2.4GHz radios (phy0-ap0), but also observed on 5GHz radios (phy1-ap0) - see statistics in attachment 01.

Configuration (relevant, sanitized)

See attachment 04 (/etc/config/wireless) and attachment 05 (/etc/config/usteer, before/after). Relevant per-interface 802.11r parameters (state with FT enabled, baseline before applying our workaround):

option ieee80211r '1'
option ft_over_ds '0'
option ieee80211k '1'
option ieee80211v '1'
option bss_transition '1'
option wnm_sleep_mode '1'
option wnm_sleep_mode_no_sched '1'
option proxy_arp '1'

Error frequency (24h observation on 25.12.4, see attachment 01)

Router	"key addition failed" count in 24h	Avg. interval
cudy1	81	~18 min
cudy2	189	~7.6 min
cudy3	383	~3.8 min

(To be re-measured on 25.12.5 with FT re-enabled; the issue was confirmed to reoccur qualitatively, exact counts pending.)

Representative log excerpt: real 3-minute connectivity outage

See attachment 02 for the full excerpt (captured on 25.12.4; same failure pattern observed for the same client after upgrading to 25.12.5). Key points:

• Client 6a:a2:5b:27:2b:37 loses its connection to phy0-ap0 at 06:57:15
• At the same moment: 14 consecutive key addition failed errors for the same client (err ... [ phy0-ap0: nl80211: kernel reports: key addition failed] + message repeated 13 times)
• Client remains completely disconnected for 3 minutes 5 seconds
• On the eventually successful reconnection at 07:00:21 (auth_alg=ft), the error occurs again immediately before the successful association - so the retry does eventually succeed, but not reliably or quickly enough to avoid a visible outage

Additional observation: possible multi-BSS relationship

See attachment 03. In several instances, the error occurs on one BSS of a radio (phy0-ap0) while an FT reassociation is happening concurrently on a different BSS of the same physical radio (phy0-ap1, a separate guest network). This suggests the race condition may not be purely per-client, but could be aggravated by contention for a shared hardware/driver resource (the radio's key table).

Elimination process – what is NOT the cause

See attachment 06 for the full timeline. In short: disabling usteer, ft_psk_generate_local, reassociation_deadline, DHCP logging (logdhcp), and switching wpad's TLS backend variant were all tested and/or ruled out (the latter because all wpad variants are built from the same source snapshot and only differ in the TLS backend, not in the FT/nl80211 code path).

Updating wpad-mbedtls to a newer snapshot build (2026.04.02) was not feasible on the stable 25.12.4 base, since the newer build is linked against newer libubus/libblobmsg-json ABI versions that are not available in the 25.12.4 repos (see the apk error captured in attachment 06).

A full point-release upgrade from 25.12.4 to 25.12.5 did not resolve the issue either - the error pattern reoccurs identically with FT re-enabled.

The decisive test: FT fully disabled

uci set wireless.default_radio0.ieee80211r='0'
uci set wireless.default_radio1.ieee80211r='0'
uci set wireless.guest_radio0.ieee80211r='0'
uci set wireless.guest_radio1.ieee80211r='0'
uci commit wireless
wifi reload

Applied on all three routers (on 25.12.4). Result: key addition failed stopped occurring entirely. We had initially disabled FT on only a single radio - the error still occurred there, because a client on another, still FT-enabled interface (same mobility domain) could still trigger an FT roam to that radio. This shows the workaround must be applied consistently across all participating interfaces to be effective.

Suspected root cause

From the mac80211 source (net/mac80211/cfg.c, function ieee80211_add_key()):

if (!sta || !test_sta_flag(sta, WLAN_STA_ASSOC)) {
    ieee80211_key_free_unused(key);
    return -ENOENT;
}

During FT, hostapd installs the PTK very tightly coupled in time to the reassociation - apparently sometimes before the internal mac80211 state transition to WLAN_STA_ASSOC has completed. The kernel then rejects the key installation with -ENOENT; hostapd logs this as key addition failed and retries after the association completes - but this retry does not always succeed reliably/quickly enough to prevent visible disconnects (see attachment 02).

Impact

Real, noticeable WiFi drops were observed on two different end-user devices (an Android smartphone and an iPhone) in everyday use, correlating in time with clusters of key addition failed events. After fully disabling FT on all routers/radios (on 25.12.4), these drops stopped occurring. After upgrading to 25.12.5 and re-enabling FT to retest, the same disconnects returned.

Current workaround

802.11r disabled on all radios/routers. This costs the small speed advantage of FT (~10-50ms) over a full reassociation (~100ms-2s); all other benefits of a multi-AP setup with shared SSID/PSK (802.11k/v, automatic AP selection via usteer, no IP change on roam) are retained.

Request

Given that the issue persists unchanged across a full point-release upgrade (25.12.4 -> 25.12.5), it does not appear to be addressed by any patches shipped in the interim. Any guidance on whether this is a known/tracked issue in mac80211 or mt76, and whether the retry-on-WLAN_STA_ASSOC behavior in ieee80211_add_key() could be revisited (e.g. deferring/queuing the key install instead of rejecting outright, as hinted at in an existing TODO comment in that function) would be very welcome.

Attachments

File	Content
00_system_info_TEMPLATE.txt	To be filled in with current system info (on 25.12.5) before posting
01_error_statistics.txt	Error frequency over 24h, all 3 routers (captured on 25.12.4)
02_log_excerpt_3min_outage.txt	Representative outage with 14 consecutive errors
03_log_excerpt_multibss.txt	Evidence of a possible multi-BSS relationship
04_wireless_config_sanitized_FT-ON.txt	Full wireless config (keys redacted), state before the workaround
05_usteer_config_before_after.txt	usteer config before/after a parameter change (included as elimination evidence)
06_troubleshooting_timeline.txt	Timeline of all tested workarounds and their results, including the 25.12.4 -> 25.12.5 upgrade test

00_system_info_TEMPLATE.txt
01_error_statistics.txt
02_log_excerpt_3min_outage.txt
03_log_excerpt_multibss.txt
04_wireless_config_sanitized_FT-ON.txt
05_usteer_config_before_after.txt
06_troubleshooting_timeline.txt

[raenye]

openwrt/openwrt#23181

[duetre2002]

Thanks @raenye, that looks like exactly the right fix for this - the PR
description mentions reproducing this specific race on Cudy WR3000P/E
(same Filogic platform as my WR3000S), and the approach (deferring the
key upload until WLAN_STA_ASSOC instead of rejecting it) matches what I
found tracing through ieee80211_add_key() in mac80211.

I'll keep an eye on #23181 and report back once it's merged and I can
test it on my setup. In the meantime I'll leave FT disabled as my
workaround (see troubleshooting timeline in the attachments above).

Is there a way to test the patch ahead of a merge - e.g. a snapshot
build or branch I could try? Happy to help verify it against my
reproduction case if useful.

[raenye]

Self build. I do this for most of my devices.
Are you familiar with git?

[duetre2002]

Not an expert, but know basics. I can follow git instructions/commands if you can
point me to the right ones (e.g. which remote/branch to fetch for
PR #23181, and any specific build config for the Filogic/mt7981
target). Happy to self-build and test against my reproduction case.

[raenye]

Setup:

git clone https://github.com/openwrt/openwrt
cd openwrt
scripts/feeds update -a
scripts/feeds install -a

Fetch and cherry-pick PR:

git fetch origin pull/23181/head
git cherry-pick FETCH_HEAD

Configure stuff (choose mediatek/filogic, choose your target, add stuff you like such as LuCI, exit and save)

make menuconfig

Build (replace 4 by number of CPU cores you have):

make -j4

Then find your sysupgrade file at bin/targets/mediatek/filogic.

EDIT: this builds snapshot, if you want to build 25.12.5 you should git checkout it after cloning.

[duetre2002]

Thanks a lot for the detailed steps, @raenye - much appreciated!

I'll run without FT for now and try building the
firmware with your instructions when I find some time. Will report back
here once I've tested it against my reproduction case.

Thanks again for the pointer to #23181 and the build walkthrough!

[duetre2002]

Update: I built OpenWrt 25.12.5 (from the openwrt-25.12 branch) with PR #23181
cherry-picked on top, and flashed it to all three of my Cudy WR3000S routers.
With FT (802.11r) re-enabled on all radios across all three devices, it has
now been running error-free for several days - zero "key addition failed"
errors, compared to the hundreds of occurrences per day I was seeing before.

This confirms the fix works well in a real multi-AP mesh setup, not just in
isolation. Thanks again @raenye and @AndreaCovelli for the fix - hoping this
can help move the PR toward a merge.