FS#896 - mt7620 - dropping frames (WiFi)

[openwrt-bot]

drut:

Device: Nexx WT3020 (MT7620)
LEDE release: LEDE Reboot 17.01-SNAPSHOT r3435-65eec8b

Latest LEDE on Nexx WT3020 with MT7620 is getting syslog spammed with:
ieee80211 phy0: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2

It usually occurs with high-bandwidth usage of WiFi client (Android i.e.) and results with need to reboot router (or WiFi won't even work for devices - it's seen but unable to connect).

# cat /etc/config/wireless

config wifi-device 'radio0'
option type 'mac80211'
option hwmode '11g'
option path 'platform/10180000.wmac'
option country 'PL'
option disabled '0'
option htmode 'HT40'
option distance '3'
option txpower '20'
option channel '6'

config wifi-iface
option device 'radio0'
option mode 'ap'
option ssid 'xxx'
option network 'lan'
option encryption 'psk2+ccmp'
option key 'xxx'

Also it's connected with the issue of android devices loosing connectivity/dropping wifi even 1 meter next to router:
Fri Jul 7 00:32:11 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: authenticated
Fri Jul 7 00:32:11 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated (aid 2)
Fri Jul 7 00:32:11 2017 daemon.notice hostapd: wlan0: AP-STA-CONNECTED xx:xx:xx:xx:xx:xx
Fri Jul 7 00:32:11 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Fri Jul 7 00:32:51 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: disconnected due to excessive missing ACKs
Fri Jul 7 00:32:51 2017 daemon.notice hostapd: wlan0: AP-STA-DISCONNECTED xx:xx:xx:xx:xx:xx
Fri Jul 7 00:32:55 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: authenticated
Fri Jul 7 00:32:55 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated (aid 2)
Fri Jul 7 00:32:55 2017 daemon.notice hostapd: wlan0: AP-STA-CONNECTED xx:xx:xx:xx:xx:xx
Fri Jul 7 00:32:55 2017 daemon.info hostapd: wlan0: STA xx:xx:xx:xx:xx:xx WPA: pairwise key handshake completed (RSN)

[openwrt-bot]

psyborg55:

[198938.155172] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2
[198938.164724] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2
[198938.174261] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2
[198938.183791] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2
[198938.193330] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2
[198938.202841] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Dropping frame due to full tx queue 2

or

[ 658.425673] ieee80211 phy1: rt2800mmio_txstatus_is_spurious: Warning - 4 spurious TX_FIFO_STATUS interrupt(s)
[ 658.465513] ieee80211 phy1: rt2800mmio_txdone: Warning - Got TX status for an empty queue 2, dropping
[ 663.563480] ieee80211 phy1: rt2800mmio_txstatus_is_spurious: Warning - 4 spurious TX_FIFO_STATUS interrupt(s)
[ 663.603233] ieee80211 phy1: rt2800mmio_txdone: Warning - Got TX status for an empty queue 2, dropping
[ 664.190083] ieee80211 phy1: rt2800mmio_txstatus_is_spurious: Warning - 4 spurious TX_FIFO_STATUS interrupt(s)
[ 664.213024] ieee80211 phy1: rt2800mmio_txdone: Warning - Got TX status for an empty queue 2, dropping
[ 683.506776] ieee80211 phy1: rt2800mmio_txstatus_is_spurious: Warning - 4 spurious TX_FIFO_STATUS interrupt(s)
[ 683.559570] ieee80211 phy1: rt2800mmio_txdone: Warning - Got TX status for an empty queue 2, dropping
[ 692.532461] ieee80211 phy1: rt2800mmio_txstatus_is_spurious: Warning - 4 spurious TX_FIFO_STATUS interrupt(s)
[ 692.637375] ieee80211 phy1: rt2800mmio_txdone: Warning - Got TX status for an empty queue 2, dropping

are not cause of interface freezing

[openwrt-bot]

psyborg55:

patch that suppose to fix this problem: https://lists.openwrt.org/pipermail/openwrt-devel/2015-September/035776.html

[openwrt-bot]

psyborg55:

as well as the guy from this issue https://bugs.lede-project.org/index.php?do=details&task_id=557 you could've write exact chipset version which in your case happens to be MT7620N

[openwrt-bot]

lugovykh:

Please fix it. I cant use my Xiaomi WiFi mini with LEDE while WiFi 2.4 is broken.

[openwrt-bot]

drut:

Is there anything going on with the issue?

[openwrt-bot]

psyborg55:

interface freeze in AP+STA mode. try scanning for APs did not work, logs revealed:

Sat Oct 7 21:33:17 2017 kern.warn kernel: [ 564.296131] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:18 2017 kern.warn kernel: [ 564.504294] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:18 2017 kern.warn kernel: [ 564.512990] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:19 2017 kern.warn kernel: [ 564.761902] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:20 2017 kern.warn kernel: [ 564.971253] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:20 2017 kern.warn kernel: [ 564.979359] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:20 2017 kern.warn kernel: [ 565.009553] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:21 2017 kern.warn kernel: [ 565.258527] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:21 2017 kern.warn kernel: [ 565.466837] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:21 2017 kern.warn kernel: [ 565.497405] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:22 2017 kern.warn kernel: [ 565.747321] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:23 2017 kern.warn kernel: [ 565.956239] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:23 2017 kern.warn kernel: [ 565.986827] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:24 2017 kern.warn kernel: [ 566.235633] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:25 2017 kern.warn kernel: [ 566.443898] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:25 2017 kern.warn kernel: [ 566.474659] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:26 2017 kern.warn kernel: [ 566.724444] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:27 2017 kern.warn kernel: [ 566.932710] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:27 2017 kern.warn kernel: [ 566.963749] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:28 2017 kern.warn kernel: [ 567.212917] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:28 2017 kern.warn kernel: [ 567.421523] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:29 2017 kern.warn kernel: [ 567.452104] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:29 2017 kern.warn kernel: [ 567.700880] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:30 2017 kern.warn kernel: [ 567.909396] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:30 2017 kern.warn kernel: [ 567.940270] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:31 2017 kern.warn kernel: [ 568.188520] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:32 2017 kern.warn kernel: [ 568.398126] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:32 2017 kern.warn kernel: [ 568.428698] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:33 2017 kern.warn kernel: [ 569.107345] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:34 2017 kern.warn kernel: [ 569.669877] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:34 2017 kern.warn kernel: [ 569.700900] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:35 2017 kern.warn kernel: [ 569.949391] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:36 2017 kern.warn kernel: [ 570.158053] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush
Sat Oct 7 21:33:36 2017 kern.warn kernel: [ 570.189239] ieee80211 phy1: rt2800_config_txpower_rt6352: Warning - ignoring EEPROM HT40 power delta: 4
Sat Oct 7 21:33:37 2017 kern.warn kernel: [ 570.678661] ieee80211 phy1: rt2x00queue_flush_queue: Warning - Queue 2 failed to flush

after re-enabling interface:

Sat Oct 7 21:35:20 2017 kern.err kernel: [ 611.146250] ieee80211 phy1: rt2x00queue_write_tx_frame: Error - Arrived at non-free entry in the non-full queue 2

[openwrt-bot]

dangowrt:

I picked up the [[https://lists.openwrt.org/pipermail/openwrt-devel/2015-September/035776.html|patch mentioned above]] with some minor fixes into [[https://git.lede-project.org/?p=lede/dangole/staging.git;a=summary|my staging tree]]. Please build, test and report back. If it improves the situation, we may carry it in our tree for a while and then push it upstream.

[openwrt-bot]

psyborg55:

patch might have fixed the issue with the current kernel version at the time it was made but not with latest 4.x.x kernel versions.

[openwrt-bot]

psyborg55:

meanwhile i found to change gpio mode:

(*&gpiomode = readl((void *)(0xb0000060)));
if ((gpiomode & 0x100000) == 0) {
gpiomode &= (~0x100000);
writel(gpiomode, (void *)(0xb0000060));

	rt2x00_info(rt2x00dev, "Change as Normal Mode(0x%lx)\n", gpiomode);
	(*&gpiomode = readl((void *)(0xb0000060)));
	rt2x00_info(rt2x00dev, "After Change GPIO_MODE value is 0x%lx\n", gpiomode);
}</code>

comp. reg has to be 0, opposed to mtk driver where it's 0x100000 representing PA gpio mode which means rt2x00 doesn't run the PA in gpio mode. mini shows the 0x48371d value with PA enabled in DTS code and 0x58371d without it enabled.

[openwrt-bot]

psyborg55:

how did you define alc for ht40 +/- 4dB ?

[openwrt-bot]

ambientsummer:

Daniel Golle, I use r5049-e53bdcb694 compiled against your tree for 4 days without any problems.
upd. Forget to mention device: xiaomi mini (MT7620A).

[openwrt-bot]

Cjcr:

Daniel Golle:

I tested last version of the staging trunk "mac80211: rt2x00: fix rt2800 queue management" doing the "git clone https://git.lede-project.org/lede/dangole/staging.git" and compiling it. The result it is that it hangs very fast and stop working but the tx queue 2 issue doens't appear this time in the logs but the result is the same. Need to restart device to make it working again.

It is setup as AP only and I use a just a laptop to connect to it. The connection hangs when reach 100-200 MB @ 3-5 MB/s. Before it only hangs when are more than one client connected. Now it seems that happens even with just one client.

Model | Nexx WT3020 (8M)
Versión del firmware | LEDE Reboot SNAPSHOT r5049-e53bdcb / LuCI Master (git-17.289.53761-19f30d0)
Versión del Kernel | 4.9.54

Edit #1: Seems that the hangs is caused just when WMM is activated in N mode, seems not related to the famous tx queue 2 issue. Maybe this is a new bug introduced.

Edit #2: For now, the __tx_queue __2 issue doesn't appear in N mode without the WMM (because if I enable this, it hangs very soon, as I said before, this sounds like a new bug). I will update this when I test it a bit more. For now looks good ...

Edit #3: 25 GB @ 3 MB/s without crash (N mode, No WMM, 2 clients = phone and laptop) :-)

[openwrt-bot]

kofec:

I have similar experience. Uptime is 2 days and till now no issues (Xiaomi MINI)
Linux version 4.9.54 () (gcc version 5.4.0 (LEDE GCC 5.4.0 r5049-e53bdcb) ) #0 Thu Oct 12 15:56:46 2017

[openwrt-bot]

Cjcr:

kofec can you try to reproduce the hang of the WiFi using N mode with WMM option on? It happens to me with high traffic load

[openwrt-bot]

psyborg55:

increasing queue limit decreases throughput. use 128/64 values, if the problem does not re-appear patch might work.