Disconnections still exist on SNAPSHOT with latest fixes. Also internet loss. MT7621|MT7603

[Openwrtfunboy]

Newifi3 D2

SNAPSHOT from 8 APR.

OpenWrt SNAPSHOT r12861-d8bde3687a / LuCI Master git-20.099.75691-88ec332

Disconnections still exists.

I sat next to the router and watched the live broadcast from YouTube.

Fri Apr 10 04:40:30 2020 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:40:31 2020 daemon.notice hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: did not acknowledge authentication response
Fri Apr 10 04:40:41 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:40:41 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:40:41 2020 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:40:41 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 WPA: pairwise key handshake completed (RSN)
Fri Apr 10 04:40:42 2020 daemon.info dnsmasq-dhcp[2650]: DHCPDISCOVER(br-lan) 00:0a:f5:9e:17:84
Fri Apr 10 04:40:42 2020 daemon.info dnsmasq-dhcp[2650]: DHCPOFFER(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:40:42 2020 daemon.info dnsmasq-dhcp[2650]: DHCPREQUEST(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:40:42 2020 daemon.info dnsmasq-dhcp[2650]: DHCPACK(br-lan) 192.168.1.232 00:0a:f5:9e:17:84 Network
Fri Apr 10 04:41:26 2020 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:41:26 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:41:33 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:41:33 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:41:33 2020 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:41:33 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 WPA: pairwise key handshake completed (RSN)
Fri Apr 10 04:41:33 2020 daemon.info dnsmasq-dhcp[2650]: DHCPDISCOVER(br-lan) 00:0a:f5:9e:17:84
Fri Apr 10 04:41:33 2020 daemon.info dnsmasq-dhcp[2650]: DHCPOFFER(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:41:33 2020 daemon.info dnsmasq-dhcp[2650]: DHCPREQUEST(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:41:33 2020 daemon.info dnsmasq-dhcp[2650]: DHCPACK(br-lan) 192.168.1.232 00:0a:f5:9e:17:84 Network
Fri Apr 10 04:43:10 2020 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:43:11 2020 daemon.notice hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: did not acknowledge authentication response
Fri Apr 10 04:43:11 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:43:11 2020 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:43:11 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 WPA: pairwise key handshake completed (RSN)
Fri Apr 10 04:43:11 2020 daemon.info dnsmasq-dhcp[2650]: DHCPDISCOVER(br-lan) 00:0a:f5:9e:17:84
Fri Apr 10 04:43:11 2020 daemon.info dnsmasq-dhcp[2650]: DHCPOFFER(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:43:11 2020 daemon.info dnsmasq-dhcp[2650]: DHCPREQUEST(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:43:11 2020 daemon.info dnsmasq-dhcp[2650]: DHCPACK(br-lan) 192.168.1.232 00:0a:f5:9e:17:84 Network
Fri Apr 10 04:43:14 2020 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:43:15 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:43:15 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:43:20 2020 daemon.notice hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: did not acknowledge authentication response
Fri Apr 10 04:43:24 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
Fri Apr 10 04:43:47 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:43:47 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:43:47 2020 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:43:47 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 WPA: pairwise key handshake completed (RSN)
Fri Apr 10 04:43:47 2020 daemon.info dnsmasq-dhcp[2650]: DHCPDISCOVER(br-lan) 00:0a:f5:9e:17:84
Fri Apr 10 04:43:47 2020 daemon.info dnsmasq-dhcp[2650]: DHCPOFFER(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:43:47 2020 daemon.info dnsmasq-dhcp[2650]: DHCPREQUEST(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:43:47 2020 daemon.info dnsmasq-dhcp[2650]: DHCPACK(br-lan) 192.168.1.232 00:0a:f5:9e:17:84 Network
Fri Apr 10 04:45:17 2020 daemon.notice netifd: wan (2386): udhcpc: sending renew to 10.8.0.1
Fri Apr 10 04:45:18 2020 daemon.notice netifd: wan (2386): udhcpc: lease of 10.8.7.99 obtained, lease time 600
Fri Apr 10 04:48:56 2020 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:48:57 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: authenticated
Fri Apr 10 04:48:57 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 IEEE 802.11: associated (aid 4)
Fri Apr 10 04:48:57 2020 daemon.notice hostapd: wlan0: AP-STA-CONNECTED 00:0a:f5:9e:17:84
Fri Apr 10 04:48:57 2020 daemon.info hostapd: wlan0: STA 00:0a:f5:9e:17:84 WPA: pairwise key handshake completed (RSN)
Fri Apr 10 04:48:57 2020 daemon.info dnsmasq-dhcp[2650]: DHCPDISCOVER(br-lan) 00:0a:f5:9e:17:84
Fri Apr 10 04:48:57 2020 daemon.info dnsmasq-dhcp[2650]: DHCPOFFER(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:48:57 2020 daemon.info dnsmasq-dhcp[2650]: DHCPREQUEST(br-lan) 192.168.1.232 00:0a:f5:9e:17:84
Fri Apr 10 04:48:57 2020 daemon.info dnsmasq-dhcp[2650]: DHCPACK(br-lan) 192.168.1.232 00:0a:f5:9e:17:84 Network

@nbd168

[Openwrtfunboy]

At the time of the tests, I noticed that when there are several clients nearby and a little bit further, then the clients who are nearby for some reason lose their Internet connection.

[Openwrtfunboy]

Also i have same on latest stable 19.07.2

[Openwrtfunboy]

Empirically, I found out that the loss of the Internet occurs even with clients connected via LAN. 4 clients were connected to the 2.4 network, 5 one was connected to the network, a laptop was connected via wire, which downloaded a 14 gigabyte file and periodically either the laptop or the rest of the clients lost the connection. Moreover, the clients were not next to the router.

[JFtico]

Also seeing similar on both 19.07.2 and snapshot r12529-228bb84744 .
See kernel log I posted in #375

[shir474]

Try moving the MT7621's switch interrupt to another core.

[JFtico]

@shir474 Thanks for the suggestion, how might I implement that?

[shir474]

see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-irq

[Openwrtfunboy]

@shir474 Thanks for the suggestion, how might I implement that?

Are there any successes, @JFtico ?

[Openwrtfunboy]

see https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-irq

There commands is not working for openwrt :(

[shir474]

It is working for me. You'll have to figure out which interrupt you need to change the smp_affinity for.

move switch interrupt to CPU2

echo 4 > /proc/irq/21/smp_affinity

move one of mt7615 interrupt to CPU1

echo 2 > /proc/irq/26/smp_affinity

cat /proc/interrupts

       CPU0       CPU1       CPU2       CPU3

8: 44463125 44463236 44463197 44463199 MIPS GIC Local 1 timer
9: 17446 0 0 0 MIPS GIC 63 IPI call
10: 0 5948 0 0 MIPS GIC 64 IPI call
11: 0 0 11662 0 MIPS GIC 65 IPI call
12: 0 0 0 5550 MIPS GIC 66 IPI call
13: 525638 0 0 0 MIPS GIC 67 IPI resched
14: 0 18989179 0 0 MIPS GIC 68 IPI resched
15: 0 0 1249439 0 MIPS GIC 69 IPI resched
16: 0 0 0 1283756 MIPS GIC 70 IPI resched
17: 0 0 0 0 MIPS GIC 19 1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
19: 4227 0 0 0 MIPS GIC 33 ttyS1
20: 0 0 0 0 MIPS GIC 29 xhci-hcd:usb1
21: 27 0 964175 0 MIPS GIC 10 1e100000.ethernet
23: 16352529 0 0 0 MIPS GIC 11 mt7615e
24: 9527128 0 0 0 MIPS GIC 31 mt7615e
27: 0 0 0 0 1e000600.gpio 17 keys
28: 0 0 0 0 1e000600.gpio 12 keys

[JFtico]

shir474, thanks for that. And I've added the echo commands to my rc.local so it applies even after a reboot.
Is there a better place to persist this in a running box ?

This is what I put in rc.local

# Set ethernet on CPU 2 (live core)
echo 4 >/proc/irq/22/smp_affinity
# Move mt76x2e 5GHz radio to CPU 1
echo 2 >/proc/irq/25/smp_affinity

I'll let it run for a few days and see how it goes.

Any idea why this works? Seems fragile.

[shir474]

It works because all the load is not on a single core. It would be better if MSI-X was implemented by the driver so that the interrupts were distributed across all cores like IPI.

PS. There's a mistake on my previous post. The second line should have been
echo 2 > /proc/irq/26/smp_affinity
Didn't realize that interrupts where moved in the latest code.

[JFtico]

Thanks, funny as on my hardware (ZBT WE3526), interrupts were already distributed, ethernet on CPU0, 5Ghz on CPU2 and 2.4 on CPU3.
Basically all I tried is moving the switch to CPU2 and 5Ghz to CPU1, so now the switch and the 2.4 share a core and 5Ghz plus whatever other interrupts are serviced by CPU0 are on the first core.
I'm thinking I was probably OK as far as core load balancing unless I'm missing something?

[Bartvz]

It works because all the load is not on a single core. It would be better if MSI-X was implemented by the driver so that the interrupts were distributed across all cores like IPI.

Which hopefully comes to mt7612/mt7603 since it came for mt7615 in these commits:

• 5f2f676
• 05b23d7

@nbd168 is that possible?