NWA50AX Pro regularily crashes during normal operation on master #2980

nrbffs · 2023-09-12T16:25:16Z

Bug report

What is the problem?

NWA50AX Pro crashes regularily (every 1-12 hours). The following stracktrace was collected in ramoops:

<1>[34681.152965] Unable to handle kernel access to user memory outside uaccess routines at virtual address 0000000000000008
<1>[34681.152984] Mem abort info:
<1>[34681.152986]   ESR = 0x0000000096000005
<1>[34681.152990]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[34681.152994]   SET = 0, FnV = 0
<1>[34681.152997]   EA = 0, S1PTW = 0
<1>[34681.153000]   FSC = 0x05: level 1 translation fault
<1>[34681.153004] Data abort info:
<1>[34681.153006]   ISV = 0, ISS = 0x00000005
<1>[34681.153009]   CM = 0, WnR = 0
<1>[34681.153013] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000041781000
<1>[34681.153018] [0000000000000008] pgd=08000000440e7003, p4d=08000000440e7003, pud=08000000440e7003, pmd=0000000000000000
<0>[34681.153032] Internal error: Oops: 96000005 [#1] SMP
<7>[34681.157901] Modules linked in: iptable_nat batman_adv xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_CT nf_nat nf_conntrack mt7915e mt76_connac_lib mt76 mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_quota xt_pkttype xt_owner xt_multiport xt_mark xt_mac xt_limit xt_comment xt_addrtype xt_TCPMSS xt_LOG ts_kmp ts_fsm ts_bm nf_reject_ipv4 nf_log_syslog nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_filter ip_tables ebtables ebt_vlan ebt_stp ebt_snat ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_ip6 ebt_ip ebt_dnat ebt_arpreply ebt_arp ebt_among ebt_802_3 compat crypto_safexcel sch_teql sch_sfq sch_multiq sch_gred sch_fq sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp act_simple act_pedit act_csum libcrc32c sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow act_skbedit act_mirred act_gact ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 dummy l2tp_eth
<7>[34681.158115]  l2tp_netlink l2tp_core ip_tunnel veth tun vxlan udp_tunnel ip6_udp_tunnel sha1_generic seqiv md5 des_generic libdes authencesn authenc leds_gpio cls_basic sch_tbf sch_ingress gpio_button_hotplug
<7>[34681.262770] CPU: 0 PID: 4484 Comm: respondd Not tainted 5.15.127 #0
<7>[34681.269024] Hardware name: ZyXEL NWA50AX Pro (DT)
<7>[34681.273714] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
<7>[34681.280662] pc : sta_set_sinfo+0xa1c/0xc40 [mac80211]
<7>[34681.285743] lr : sta_set_sinfo+0x5bc/0xc40 [mac80211]
<7>[34681.290806] sp : ffffffc00956b5e0
<7>[34681.294108] x29: ffffffc00956b5e0 x28: 0000000000000001 x27: ffffffc00956bdd0
<7>[34681.301233] x26: ffffff8000064880 x25: ffffff8002938900 x24: ffffff8005f3e735
<7>[34681.308358] x23: 00000000000023a2 x22: 0000000000000004 x21: 0000058d1783bf82
<7>[34681.315483] x20: ffffff8005f3e000 x19: ffffffc00956b728 x18: 0cea122a0dac8927
<7>[34681.322607] x17: 000000000b9f76c0 x16: 0675091506d65a47 x15: 0000000005cfbb60
<7>[34681.329733] x14: 0000000000000000 x13: 000000000206cc80 x12: 010f4471011f2ba0
<7>[34681.336857] x11: 0000000000f42400 x10: 00000000000000a7 x9 : 0000000000000000
<7>[34681.343981] x8 : ffffffc00956b810 x7 : ffffff8002aa0880 x6 : 000000000000003f
<7>[34681.351105] x5 : ffffff8002aa03a0 x4 : 0000000000000000 x3 : 0000000000000000
<7>[34681.358229] x2 : ffffffc00956b76c x1 : 0000000000000001 x0 : 0000000000000001
<7>[34681.365354] Call trace:
<7>[34681.367788]  sta_set_sinfo+0xa1c/0xc40 [mac80211]
<7>[34681.372504]  ieee80211_color_change_finish+0x1b64/0x1e80 [mac80211]
<7>[34681.378781]  nl80211_put_sta_rate+0x1030/0x11b4 [cfg80211]
<7>[34681.384270]  netlink_dump+0x104/0x360
<7>[34681.387925]  __netlink_dump_start+0x158/0x2ac
<7>[34681.392271]  genl_family_rcv_msg_dumpit+0x64/0xf4
<7>[34681.396963]  genl_rcv_msg+0x188/0x1c0
<7>[34681.400614]  netlink_rcv_skb+0x58/0x120
<7>[34681.404440]  genl_rcv+0x34/0x50
<7>[34681.407571]  netlink_unicast+0x1f0/0x2ec
<7>[34681.411483]  netlink_sendmsg+0x19c/0x3d0
<7>[34681.415396]  ____sys_sendmsg+0x27c/0x2b0
<7>[34681.419311]  ___sys_sendmsg+0x80/0xf0
<7>[34681.422963]  __sys_sendmsg+0x44/0xa0
<7>[34681.426527]  __arm64_sys_sendmsg+0x20/0x30
<7>[34681.430611]  invoke_syscall.constprop.0+0x4c/0xe0
<7>[34681.435305]  do_el0_svc+0x40/0xd0
<7>[34681.438609]  el0_svc+0x14/0x50
<7>[34681.441656]  el0t_64_sync_handler+0xe0/0x110
<7>[34681.445916]  el0t_64_sync+0x15c/0x160
<0>[34681.449570] Code: d3441c63 12000c00 8b030ca3 f9409c63 (f9400465) 
<4>[34681.455650] ---[ end trace 3a7b82ed9e2e38bf ]---
<0>[34681.465492] Kernel panic - not syncing: Oops: Fatal exception
<2>[34681.471228] SMP: stopping secondary CPUs
<0>[34681.475142] Kernel Offset: disabled
<0>[34681.478617] CPU features: 0x0,00000000,20000802
<0>[34681.483135] Memory Limit: none

What is the expected behaviour?
Does not crash.

Gluon Version:
8c48f20c5df80fa1f038c0c69edda61213b4eb6a

Site Configuration:

https://gitlab.freifunk-stuttgart.de/firmware/site-ffs
Issue observed on https://gitlab.freifunk-stuttgart.de/firmware/site-ffs/-/commit/397b6f9b8a8de4cb98e45f74203632c4004ad882

Custom patches:

The text was updated successfully, but these errors were encountered:

neocturne · 2023-09-12T16:40:41Z

Can you recheck with latest master? There has been an OpenWrt update that includes an mt76 bump (although I'm unsure if this is specific to mt76, as it doesn't appear in the backtrace)

nrbffs · 2023-09-12T16:50:44Z

Node is now running 251392b026c8be11880797dd8c70a8ecd2f3c0b6: http://[2a01:1e8:c003:930b:f64d:5cff:fe5a:93ce]/cgi-bin/status
I will report back if the issue persists.

nrbffs · 2023-09-14T15:13:42Z

I applied the following patch to OpenWRT:

diff --git a/package/kernel/mac80211/patches/build/900-ffs-debug.patch b/package/kernel/mac80211/patches/build/900-ffs-debug.patch
new file mode 100644
index 0000000000..d2efe966d1
--- /dev/null
+++ b/package/kernel/mac80211/patches/build/900-ffs-debug.patch
@@ -0,0 +1,13 @@
+--- a/net/mac80211/sta_info.c
++++ b/net/mac80211/sta_info.c
+@@ -2355,6 +2355,10 @@ static void sta_stats_decode_rate(struct
+ 
+               sband = local->hw.wiphy->bands[band];
+ 
++              printk("sband = %p, band = 0x%x\n", sband, band);
++              if (!sband)
++                      break;
++
+               if (WARN_ON_ONCE(!sband->bitrates))
+                       break;
+

This works around the crashes, but sband still is sometimes zero:

root@71522-amschillerplatz9-node-02:~# dmesg|grep -E 0{16}
[64609.392171] sband = 0000000000000000, band = 0x0
[66459.862191] sband = 0000000000000000, band = 0x1
[68653.212194] sband = 0000000000000000, band = 0x1
[69183.642352] sband = 0000000000000000, band = 0x0

This is based on gluon commit 251392b026c8be11880797dd8c70a8ecd2f3c0b6

blocktrron · 2023-09-19T20:19:54Z

I've tried to reproduce your issue but without luck.

Can you add to your workaround if local->hw.wiphy and local itself is already zero?

To me it looks like a lock is not acquired and somehow we end up in a race-condition. The calltrace itself also does not look linear.

nrbffs · 2023-09-22T14:37:35Z

I've tried to reproduce your issue but without luck.

Can you add to your workaround if local->hw.wiphy and local itself is already zero?

Only sband seems to be NULL:

[15051.211939] local = 00000000ae390e6d, local->hw.wiphy = 00000000283f497b, sband = 0000000000000000, band = 0x0

nrbffs · 2023-11-12T08:19:53Z

Anything I can provide or help with? Should I make a PR with the workaround?

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Closes: freifunk-gluon#2980 Link: freifunk-gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

blocktrron · 2023-12-03T06:14:03Z

@nrbffs Please see #3084

rotanid · 2023-12-03T13:47:51Z

@blocktrron should this really be closed by #3084 , is there some issue (upstream?) tracking this?

blocktrron · 2023-12-03T13:49:56Z

I don't think there's an upstream ticket yet.

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the root cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net> (cherry picked from commit 1278d47)

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Djfe · 2024-02-03T18:07:51Z

Just to confirm:
The Netgear WAX220 is also affected:
https://paste.debian.net/hidden/cef51d2c/

EDIT:
This is on commit
bfbefa4

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net> (cherry picked from commit 1278d47)

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Closes: freifunk-gluon#2980 Link: freifunk-gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net>

Djfe · 2024-06-13T13:58:53Z

The Cudy WR3000 is also affected

Frequent crashes have been observed on MT7916 based platforms. While the root of these crashes are currently unknown, they happen when decoding rate information of connected STAs in AP mode. The rate-information is associated with a band which is not available on the PHY. Check for this condition in order to avoid crashing the whole system. This patch should be removed once the roout cause has been found and fixed. Link: freifunk-gluon/gluon#2980 Signed-off-by: David Bauer <mail@david-bauer.net> (cherry picked from commit 1278d47beaabaa963b2956e81936269b7fea4003)

neocturne added this to the v2023.2 milestone Oct 10, 2023

neocturne added the 0. type: bug This is a bug label Oct 10, 2023

blocktrron mentioned this issue Dec 3, 2023

mac80211: avoid crashing on invalid band info #3084

Merged

rotanid closed this as completed in #3084 Dec 3, 2023

blocktrron mentioned this issue Dec 7, 2023

mac80211: avoid crashing on invalid band info openwrt/openwrt#14162

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NWA50AX Pro regularily crashes during normal operation on master #2980

NWA50AX Pro regularily crashes during normal operation on master #2980

nrbffs commented Sep 12, 2023

neocturne commented Sep 12, 2023

nrbffs commented Sep 12, 2023 •

edited

Loading

nrbffs commented Sep 14, 2023

blocktrron commented Sep 19, 2023

nrbffs commented Sep 22, 2023

nrbffs commented Nov 12, 2023

blocktrron commented Dec 3, 2023

rotanid commented Dec 3, 2023

blocktrron commented Dec 3, 2023

Djfe commented Feb 3, 2024 •

edited

Loading

Djfe commented Jun 13, 2024

NWA50AX Pro regularily crashes during normal operation on master #2980

NWA50AX Pro regularily crashes during normal operation on master #2980

Comments

nrbffs commented Sep 12, 2023

Bug report

neocturne commented Sep 12, 2023

nrbffs commented Sep 12, 2023 • edited Loading

nrbffs commented Sep 14, 2023

blocktrron commented Sep 19, 2023

nrbffs commented Sep 22, 2023

nrbffs commented Nov 12, 2023

blocktrron commented Dec 3, 2023

rotanid commented Dec 3, 2023

blocktrron commented Dec 3, 2023

Djfe commented Feb 3, 2024 • edited Loading

Djfe commented Jun 13, 2024

nrbffs commented Sep 12, 2023 •

edited

Loading

Djfe commented Feb 3, 2024 •

edited

Loading