MT7603 - persistent issues #167
Comments
|
Update: Issues still happening on e2eedc9. There is a definite improvement though - it is self-recovering now. Typically, it will recover in about ~30 seconds. Just like before, station dump shows odd behavior during the connection issues. The three dumps below were taken consecutively over 3 seconds, and you can see the rx bitrate bounce around. root@ph:/etc/config# iw dev wlan0 station dump |
|
I'm seeing the same behaviour as CR-RYAN, using a Xiaomi Router 3G. Some time after connecting (few minutes) loading web sites becomes very slow, and eventually none of them load anymore. Weirdly enough some existing connections appear to keep functioning during this behaviour. E.g. I can access LUCI web interface, and if I start streaming Live TV right after connecting, I can continue streaming without a problem. Like with CR-RYAN, setting HTMODE to NONE, effectively disabling Wireless N, gets rid of the problem. Also I experience none of these problems on my 5GHZ (MT7612). The Wireless config for 2.4Ghz, that gives me the problems is as follows: |
|
I have updated to OpenWrt SNAPSHOT r6772 and now I have the same problem with iPad connected to 2.4 GHz. |
|
As not said, option disassoc_low_ack 0 did not solve the problem. |
|
@Mafesa - We are seeing the same thing - using the disassoc_low_ack 0 option doesn't fix the issue for us either. |
|
@CR-Ryan I do not know if it's a coincidence, maybe nbd knows why, but I have updated to r6787, set option lecacy to 0 and disabled Flow Offload. |
|
@Mafesa - I just tried "option legacy_rates '0' (and "1"), and neither seemed to fix the issue for me. I hadn't heard of Flow Offload before, and couldn't find anything on the wiki about it. Is that a wireless option? |
|
@CR-Ryan I think he means this: https://forum.lede-project.org/t/xiaomi-wifi-router-3g/5377/696 . Its a firewall feature. I continue to have the problems, whether its on or off. Although I had it switched on in my most recent tests. If I have time, I'll try it in combination with legacy_rates 0 soon, to see if it has any effect for me... |
|
@Tom-Brouwer Thanks for that - it looks like that feature is off by default, but I set it to 0 just to be safe. Made no difference, issues persist. I just noticed the config you posted was missing this option: option disassoc_low_ack '0' Have you tried running with that set to zero? |
This morning router reboot itself and now is very unstable, 2.4 GHz it's not usable.
Do not waste time, it's useless. Does anyone remember when the problems started? |
|
We haven't been able to use it on ht40 for quite some time
…On Wed, May 2, 2018, 9:37 AM MaFeSa ***@***.***> wrote:
For now I have no problems for 22 hours with 10 clients connected.
This morning router reboot itself and now is very unstable, 2.4 GHz it's
not usable.
Have you tried running with that set to zero?
@CR-Ryan <https://github.com/CR-Ryan> @Tom-Brouwer
<https://github.com/Tom-Brouwer>
Do not waste time, it's useless.
Does anyone remember when the problems started?
With which commit?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#167 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB0Tcncl3l7Ixfgx5b0E4D5-LKJrsFfNks5tueDagaJpZM4TY3Uq>
.
|
|
I'll try to figure out where the problem is, I start testing the commits since April 12th. |
|
@CR-Ryan I'm not sure, but I think I tried option disassoc_low_ack '0' before. I'll try to test that this weekend. @slthomason For me, this issue also occurs with HTMODE set to 20... |
|
I tried HT20 and HT40 with no results, the only way for a stable 2.4 GHz connection is to set up "Lecacy" with LuCI and disable HT mode. My stable config for 2.4 |
|
@CR-Ryan and I have found the same thing. It has been this way for some
time and this latest release is the first to make it better, but still not
totally working.
…On Fri, May 4, 2018 at 6:09 AM, MaFeSa ***@***.***> wrote:
I tried HT20 and HT40 with no results, the only way for a stable 2.4 GHz
connection is to set up "Lecacy" with LuCI.
My config for 2.4
config wifi-device 'radio0'
option type 'mac80211'
option hwmode '11g'
option path 'pci0000:00/0000:00:00.0/0000:01:00.0'
option channel '1'
option country '00'
option legacy_rates '1'
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#167 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB0Tcp6kqd3ANnPOD_o7tKn-hAmAV6xDks5tvFMKgaJpZM4TY3Uq>
.
|
|
@Tom-Brouwer Yeah, HT20 has the same issues. We have been fighting these issues for over a year now. @Mafesa The legacy option didn't do anything for us, but setting the HTmode to none does fix it. |
|
@CR-Ryan |
I have this problem only from April 12th. |
|
@Mafesa Ok - thx for clarifying on the legacy option We first started using MT7603 hardware in April of last year, and had problems from day 1. The HTmodes didn't work correctly back then either. It is definitely better now, but it is still not 100% stable on 7603. We also tested the stock firmware provided by the manufacturer of our equipment, and the drivers they are using work fine. The issue is they are on a really old kernel, so we can't use their firmware. |
|
Yes, I'm also using mt7603e (I think) on VoCore2 and I have packet loss issues (using only adhoc), so if the fix for #152 by @nbd163 also affects mt7603 it would be great news! |
|
I ran some tests, MT7603 does not have the same issue |
|
Hrm. It very very similar around devices coming out of power save mode.
What do you suggest for this one? What more data do you want to see?
…On Fri, May 18, 2018, 9:56 AM nbd168 ***@***.***> wrote:
I ran some tests, MT7603 does not have the same issue
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#167 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB0TculmfnmUeRFrYLvh6oGXM7OrfiZ5ks5tzv0xgaJpZM4TY3Uq>
.
|
|
You could make a pcap of the issue in action, with a separate device for monitoring |
|
Thanks for the suggestion. Will do.
…On Fri, May 18, 2018 at 10:34 AM, nbd168 ***@***.***> wrote:
You could make a pcap of the issue in action, with a separate device for
monitoring
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#167 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB0TcnfBRXY6_7r_IdqRhJnxenUfsWXgks5tzwY5gaJpZM4TY3Uq>
.
|
|
@nbd168 - Here is our first stab at a packet capture. The tcpdump below was taken while a device was in a "bad state". It was connected to the AP, but couldn't browse or ping the AP. tcpdump -vv host android-7638b9f93e1c93d0 -i wlan0 > /etc/tcpdump 13:25:49.688688 ARP, Ethernet (len 6), IPv4 (len 4), Reply is-at 78:a3:51:2d:a1:26 (oui Unknown), length 28 I only know of two ways to capture packets: tcpdump and wireshark. Is there a better way for us to do this? I know wireshark can capture radio layer packets, but that requires "monitor mode", which I have not been able to enable. I don't think my devices wireless driver supports it. Please let us know if there is any other info we can grab for you - we are anxious to fix this - we have thousands of customers impacted by this bug. We can easily reproduce this when using any htmode that is not "NONE". (So HT20/40) We are totally willing to spin up remote access to both the router AND the affected device during one of these issues if that would help you. We know you are busy, and so we appreciate any time you can spend on this. In fact, we would be more than happy to buy you pizza + a sixer of your favorite beer if you can fix this. Seriously. : ) |
|
Only real monitor mode captures are helpful for debugging this. The output should be a real .pcap file. By the way, I've pushed some more fixes for MT7603 and MT7628, please test the latest version from mt76.git (not pushed to OpenWrt yet). |
|
Hello, I also experience similar problems with MT7603. The issues start immediately after connecting. As has been mentioned; these problems go away when setting htmode to none. I am using a Xiaomi 3G router with openwrt master and the latest mt76 driver: 792dbe0. I have captured WLAN frames using the WLAN card of a second device in monitor mode: mt7603.pcapng is a capture when htmode is set to HT40 and running a speedtest. Initially there is no internet connection, after disconnecting from the AP and reconnecting a connection is possible but the speeds fluctuate and then go down to no connectivity yet again. mt7603_htmode_none.pcapng is a capture when htmode is set to none and running the same speedtest. I experienced no connectivity or stability problems at all.
(you can use the following display filter with wireshark to hide view obstructing beacon frames: wlan.fc.type_subtype != 0x08 ) Hopefully these captures help to solve these issues. |
|
We just pulled the latest OpenWRT code for 18.06, and are having build issues. Both wireless radios won't come up, and we get some nasty error messages. Is anyone else seeing this? |
|
VoCore2 with mt7603e running OpenWrt SNAPSHOT, r7050-9c409cb: |
|
Ah, ok, it seems to happen with other wifi too, tried with RT5370 against RTL8192CU, at first anyway: But wait, there's more, after I took mt7602e interfaces down (ifconfig down wlan0) and waited a bit, the results on the other usb wifi devices got better, at least part of the time: The RT5270 host had some of this in the logs though: So I swapped the RT5270 for a RTL8188CUS and it seems to talk to RTL8192CU quite ok right away: Then going back to the mt7603e to mt7603e I get very little again. All devices being tested at the same adhoc network: I also have batman set up among all the interfaces, but for these test I used the interfaces directly. |
|
Just saw this immediately after reboot: |
|
Update: Fortunately, a fix was pushed earlier today and wireless is working correctly again. However, we are still seeing instability on 7603, when using any htmodes. This is on the latest commit. (792dbe0) |
|
I also confirm that on openwrt commit 6c81c27 things are back to "normal" - mt7603e wifi works somewhat, but not enough to maintain a stable ssh connection, for example, even with htmode NONE. I run IBSS only with batman on top. If I switch to known-good USB sticks everything works ok while keeping everything else the same. |
|
I find the output of tcpdump on wlan0 interesting. When the wireless interface becomes unstable, EAPOL and DHCP work fine, but mDNS and ARP only work in one direction - from the wireless client to the rest of the network. The ARP respsonses from the LAN network do not show up on the wlan0 tcpdump but do on br-lan. Something is dropping the ARP traffic, possibly before the wireless driver can process it. I tested this on an MT7620 which I believe uses the rt2800pci driver and not mt76. Setting htmode to none didn't help in my case. |
|
I found out, that IEEE80211_TX_STAT_ACK can be set even with MT_TXS0_ACK_TIMEOUT set. I don't know if repairing it could help, but could you try with replacing if (!fixed_rate && !ack_timeout) With if (!fixed_rate && !ack_timeout) { ? |
|
Current state at openwrt 7590c3c is that if I have just one VoCore2 mt7603e device in my mesh (adhoc with batman on top, HT NONE) things seem to be quite ok, at least initially. However, if I add another mt7603e device things get hairy. New device won't work at all and the one next to it that was ok just a moment ago goes bad as well. I can have many devices in the same mesh with non-mt76 chips, but only one mt7603e at a time. Don't know if this helps at all, but that's my observation as of 7590c3c. |
|
This is what happens when I have a ping against 8.8.8.8 and a second mt7603e device joins the same adhoc+batman network: The pings will get even higher and I have reboot the first device before it works again: After reboot that single mt7603e device is happy again - as long as it's the only one: |
|
from my study into mt7603en it seems this is not purely a software/driver issue, but a hardware one, in short, this 2.4Ghz chip itself is not robust, a router replacement(that does not use mt7603en for 2.4Ghz) is the only dependable fix. |
|
No we have gotten it much more stable and are continuing to make
improvements on it.
…On Wed, Jul 25, 2018 at 10:47 AM laoshaw ***@***.***> wrote:
from my study into mt7603en it seems this is not purely a software/driver
issue, but a hardware one, in short, this 2.4Ghz chip itself is not robust,
a router replacement(that does not use mt7603en for 2.4Ghz) is the only
dependable fix.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#167 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AB0TcoCnTqJE7X0I7XOAoEfL0GipDY7Wks5uKK8UgaJpZM4TY3Uq>
.
|
|
running newest 18.06.1 on zbt1326, 2.4Ghz will still fail often, with htmode set to either HT20 or none(legacy 11bg), to the point not usable, use legacy alone(11bg) will not give me stable connection. another non-mt7603e 2.4G router running 18.06.1 stays connected 24x7 all the time. |
|
Which git revision are you running? |
|
For openwrt 18.06 git: for mt76: |
|
I draw a live graph to monitor /proc/net/wireless and ping at the PC in parallel, in the background I download some data from the router constantly to generate the traffic for my testing. My router is zbt1326 which has mt7603en for 2.4G. Final testing result: For both STAs they lose network 3~4 time per day(wifi signal stays, just no network), roughly 1 of them will never resume the network connection, I had to re-connect to the Openwrt to get both wifi associated and network working. In short, mt7603en AP has issues when 'wl' STA is used, it works well with ath9k STA. 'wl' STA works well with atk9k AP however. |
|
Please test the latest version from OpenWrt master or the 18.06 branch. |
|
with brcmsmac which had not even a single issue for weeks with a tplink C7 running 18.06, I just upgraded the newest 18.06 build to a zbt1326 and connected via the same brcmsmac STA, the network(not the wireless) is gone after 20 minutes(downloading traffic at the background via 2.4Ghz), and it does not recover(for another 20 minutes). the wifi speed is 5Mbps with wget for a 50MB image file. on another ath9k STA laptop now it's running 40 minutes and the connection stays solid, with downloads at 24Mbps via the same 2.4Ghz(channel 11 which is not crowded here), everything else as far as network configuration etc is the same for both test scenarios, only difference is the STA. In short, zbt1326 still is not robust, especially when I used it with brcmsmac STA, it is probably worse now, as in previous tests I did not lose network this quickly and if it's lost, most of the time it can recover. |
|
I need to know, which of the last commits made it worse for you. Please try this: Edit package/kernel/mt76/Makefile, set PKG_SOURCE_VERSION to these values, one by one, in order, and run make (no make clean required) and test again with the brcmsmac sta after each change: Please stop at the first hash value that makes it work again, and let me know which one that was. Thanks |
|
run each for 2+ hours, not much difference for them: all lost network(not wireless) 2~3 times, all caused reboot one or two times(not sure this is totally wireless related though), 6e18(first four digits in the hash) had one case that failed to self-recover network connection. all download speed is about 6Mbps on average(which is much slower comparing to ath9k STA). For 7daf I got a kernel hang within 10 minutes and I ended up power-cycling the device, also with 7daf my PC/STA can no longer have wireless work, I had to reboot my PC and reload it with "modprobe wl", looks like the STA has some issues too, but I never had problems with other openwrt routers(e.g. tplink) in the last few years while using the same PC. I will hook a serial cable and do more tests later, the above tests are preliminary. The STA is a ubuntu 18.04 with brcmsmac driver. Using wl driver instead of brcmsmac(since brcmsmac was totally dead once and ubuntu 18.04 default to wl anyways), I captured the first kernel log: This is with 7daf962 build, I have seen similar reboot in the past so it might not be that sensitive to these SHA1 checkpoints. The reboot seems triggered by the download-over-wifi-2.4Ghz-zbt1326. I have 3 ZBT1326 routers all have the same issues. second kernel reboot log, same software: Another one, this one is a default built, means I used the default profile without any other packages/modules, I used 5 scp streams(download and upload) to stress zbt1326 instead of the one-way-wget method, this crash happens fast. Now this becomes a zbt1326 problem, it keeps rebooting: |
|
Use ath9k STA for overnight iperf3 stressing test, one reboot and one self-recover network connection observed, otherwise the connection remains solid, the 7603e wireless problem is more related to broadcom STAs in my case. When using Archer C7 18.06 router with both STAs(bcm43xx and ath9k), they are 100% solid for 10+ hours, no loss of connection, no reboot,etc |
|
Should be fixed in current versions |
After pulling the latest commit (ca5cc9a), we are still seeing instability on 7603
Behavior is almost identical to what was described in this post
The device I am currently testing with is an LG v20, uses a dual band 11ac chipset. We have seen this same behavior across many other devices though - it's not just this one.
After initially connecting, the device remains connected for about 1 - 10 minutes. During which browsing and speedtests work fine. Eventually, the wireless connection gets "stuck". The device shows as connected to the network, but it cannot ping the AP, and the AP can't ping it. Obviously browsing doesn't work in this state. Disconnecting and reconnecting will typically fix this temporarily. Also, setting htmode to 'none' appears to prevent the issue entirely - but this is not a great fix.
There is interesting behavior showing in the station dumps. When in this "stuck" state, the rx bitrate will continuously jump between 24Mbits and 144Mbps. The two station dumps below were taken during a speedtest, 1 second apart, and displays this behavior:
root@ph:/etc/config# iw dev wlan0 station dump
Station a8:b8:6e:7d:06:fe (on wlan0)
inactive time: 110 ms
rx bytes: 1102876
rx packets: 11322
tx bytes: 26489082
tx packets: 9515
tx retries: 333
tx failed: 9
rx drop misc: 7
signal: -46 [-47, -46] dBm
signal avg: -47 [-49, -47] dBm
tx bitrate: 144.4 MBit/s MCS 15 short GI
rx bitrate: 24.0 MBit/s
expected throughput: 46.875Mbps
authorized: yes
authenticated: yes
associated: yes
preamble: short
WMM/WME: yes
MFP: no
TDLS peer: no
DTIM period: 2
beacon interval:100
short preamble: yes
short slot time:yes
connected time: 134 seconds
root@ph:/etc/config# iw dev wlan0 station dump
Station a8:b8:6e:7d:06:fe (on wlan0)
inactive time: 60 ms
rx bytes: 1102964
rx packets: 11324
tx bytes: 26489144
tx packets: 9516
tx retries: 333
tx failed: 9
rx drop misc: 7
signal: -53 [-54, -53] dBm
signal avg: -48 [-49, -48] dBm
tx bitrate: 144.4 MBit/s MCS 15 short GI
rx bitrate: 144.4 MBit/s MCS 15 short GI
expected throughput: 46.875Mbps
authorized: yes
authenticated: yes
associated: yes
preamble: short
WMM/WME: yes
MFP: no
TDLS peer: no
DTIM period: 2
beacon interval:100
short preamble: yes
short slot time:yes
connected time: 135 seconds
This is in contrast to when it is working well, where the rx bitrate stays on 144Mbps continuously. Here is a station dump from when it was working fine:
Station a8:b8:6e:7d:06:fe (on wlan0)
inactive time: 10 ms
rx bytes: 1085972
rx packets: 11015
tx bytes: 26449459
tx packets: 9430
tx retries: 253
tx failed: 5
rx drop misc: 3
signal: -49 [-49, -51] dBm
signal avg: -47 [-47, -50] dBm
tx bitrate: 144.4 MBit/s MCS 15 short GI
rx bitrate: 144.4 MBit/s MCS 15 short GI
expected throughput: 87.158Mbps
authorized: yes
authenticated: yes
associated: yes
preamble: short
WMM/WME: yes
MFP: no
TDLS peer: no
DTIM period: 2
beacon interval:100
short preamble: yes
short slot time:yes
connected time: 22 seconds
Here is our wireless config:
config wifi-device 'radio0'
option type 'mac80211'
option hwmode '11g'
option path 'pci0000:00/0000:00:01.0/0000:02:00.0'
option disabled '0'
option channel '6'
option country 'US'
option txpower '23'
option noscan '1'
option htmode 'HT40'
option log_level '1'
config wifi-iface 'default_radio0'
option device 'radio0'
option network 'lan'
option mode 'ap'
option hidden '0'
option disassoc_low_ack '0'
option ssid 'CleanRouter'
option encryption 'psk2'
option key '3453453456'
config wifi-device 'radio1'
option type 'mac80211'
option hwmode '11a'
option path 'pci0000:00/0000:00:00.0/0000:01:00.0'
option htmode 'VHT80'
option channel '149'
option country 'US'
option txpower '30'
option noscan '1'
option disabled '0'
option log_level '1'
config wifi-iface 'default_radio1'
option device 'radio1'
option hidden '0'
option encryption 'psk2'
option network 'lan'
option ssid 'CleanRouter5GHz'
option disassoc_low_ack '0'
option mode 'ap'
option key '3453453456'
Dmesg and logread look fine - no errors present. This is true even when the connection gets "stuck".
I can reproduce this reliably - I am happy to provide more info if needed.
Thanks!
The text was updated successfully, but these errors were encountered: