QCA9980, r7500v2 with openwrt & ath10k-ct driver & fw 10.4b-ct-9980-fH-012-620ae9

[ghost]

Hi,

I hope this is the kind of feedback you are looking for regarding the December 2018 updates to the ath10k-ct driver/firmware.

Looks like the fw crashed just after boot for both phy0 and phy1 and then phy0 crashed a few times after that. It seems like fw restarted and router/fw is still going for now.

dmesg and other output is here.

I (still) experience unexplained dropping android and iphone clients but not as frequently as with the prior ath10k-ct driver/firmware (both htt or non-htt). After reading this, I plan to try the suggestion in the "Disable/tune firmware station-kickout" sections of your user guide (assuming I'm making the right inferences).

Unexplained mobile client drops also happens with the "stock" ath10k driver/firmware but its too soon to say if this ct update is better than than stock. Devices that don't put their wifi to sleep seem to be ok so far.

I also get station drops with GTK rekey events with stock and prior ct driver/fw's. I haven't tested if there is any change yet with this ct iteration.

The crashing (router rebooting) every 12-24 hours or so I experienced with the prior ct driver/firmware have not reoccurred yet so thank you for that.

I've had the r7500v2 running openwrt for less than a month (home use). I'm less than impressed with Qualcom/atheros. I hold out hope for Candela but I suspect poor hardware design/quality contributes to some of the symptoms I exprience and read about.

Moving on from complaining...

I have one r7500v2 running OpenWrt SNAPSHOT, r8755-74a03f5f01 built from master with this mod to the dts file (increase the overlay partition to 79M.) I hope this doesn't matter...

The openwrt image is built with the "stock" ath10k driver and firmware built in. After installing the image, I removed the stock ath10k driver and firmware and installed the 2018-12-11 updated ath10k-ct driver and updated firmware with:

cp /tmp/QCA99X0-firmware-5-ct-full-htt-mgt-community-12.bin-lede.001 /lib/firmware/ath10k/QCA99X0/hw2.0/ct-firmware-5.bin

and

cp /tmp/QCA988X-firmware-2-ct-full-htt-mgt-community-22.bin.lede.001 /lib/firmware/ath10k/QCA988X/hw2.0/ct-firmware-5.bin

then rebooted. Given that:

root@OpenWrt:~# ethtool -i wlan0
driver: ath10k_pci
version: 4.14.88
firmware-version: 10.4b-ct-9980-fH-012-620ae964
expansion-rom-version: 
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

and

root@OpenWrt:~# ethtool -i wlan1
driver: ath10k_pci
version: 4.14.88
firmware-version: 10.4b-ct-9980-fH-012-620ae964
expansion-rom-version: 
bus-info: 0001:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

both show 10.4b-ct-9980-fH-012-620ae964 as the fw, I'm confused about if I need the QCA988X fw... I did it because openwrt's .config includes both fw's by default for the r7500v2. Maybe I'll delete the QCA988x and see if 2.4 GHz still works.

Both the 2.4 GHz (phy1) and the 5 GHz (phy0) radios are in use and both have a guest virtual AP (different SSIDs for all). Redacted hostapd.conf files below. IPv6 with a 60 prefix delegated to 64's for
the "guest" and "lan" interfaces is used.

It seems that irqbalance helps with the r7500v2 and ath10k - it is started early in the boot process (because I can, I don't know if I should).

The "wol" stuff in dmesg is a custom script to wakeonlan a lightly used www sever (and likely soon to be my next router).

a few static address, some port forwards to the www server, opendns, ddns, etc. etc. nothing fancy with the firewall except the wol stuff.

I also change the mac on the WAN interface at boot with an init script because I'm too impatient for my ISP to figure out that I play around with routers occasionally. I hope this does not matter...

cat /var/run/hostapd-phy1.conf
driver=nl80211
logger_syslog=127
logger_syslog_level=2
logger_stdout=127
logger_stdout_level=2
country_code=US
ieee80211d=1
hw_mode=g
beacon_int=100
channel=11


ieee80211n=1
ht_coex=0
ht_capab=[LDPC][SHORT-GI-20][SHORT-GI-40][TX-STBC][RX-STBC1][MAX-AMSDU-7935][DSSS_CCK-40]

interface=wlan1
ctrl_interface=/var/run/hostapd
ap_isolate=1
ap_max_inactivity=3600
bss_load_update_period=60
chan_util_avg_period=600
disassoc_low_ack=0
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
utf8_ssid=1
wpa_group_rekey=86400
wpa_passphrase=wouldn'tyouliketoknow
auth_algs=1
wpa=2
wpa_pairwise=CCMP
ssid=WLAN24
bridge=br-lan
wpa_disable_eapol_key_retries=0
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=no:ta:re:al:ma:c


bss=wlan1-1
ctrl_interface=/var/run/hostapd
ap_isolate=1
ap_max_inactivity=3600
bss_load_update_period=60
chan_util_avg_period=600
disassoc_low_ack=0
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
utf8_ssid=1
wpa_group_rekey=86400
wpa_passphrase=noonecares
auth_algs=1
wpa=2
wpa_pairwise=CCMP
ssid=guest24
bridge=br-guestWLAN
wpa_disable_eapol_key_retries=0
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=no:ta:re:al:ma:c

root@OpenWrt:~# cat /var/run/hostapd-phy0.conf 
driver=nl80211
logger_syslog=127
logger_syslog_level=2
logger_stdout=127
logger_stdout_level=2
country_code=US
ieee80211d=1
ieee80211h=1
hw_mode=a
beacon_int=100
channel=36


ieee80211n=1
ht_coex=0
ht_capab=[HT40+][LDPC][SHORT-GI-20][SHORT-GI-40][TX-STBC][RX-STBC1][MAX-AMSDU-7935][DSSS_CCK-40]
vht_oper_chwidth=1
vht_oper_centr_freq_seg0_idx=42
ieee80211ac=1
vht_capab=[RXLDPC][SHORT-GI-80][TX-STBC-2BY1][SU-BEAMFORMER][SU-BEAMFORMEE][MU-BEAMFORMER][MU-BEAMFORMEE][RX-ANTENNA-PATTERN][TX-ANTENNA-PATTERN][RX-STBC-1][MAX-MPDU-11454][MAX-A-MPDU-LEN-EXP7]

interface=wlan0
ctrl_interface=/var/run/hostapd
ap_isolate=1
ap_max_inactivity=3600
bss_load_update_period=60
chan_util_avg_period=600
disassoc_low_ack=0
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
utf8_ssid=1
wpa_group_rekey=86400
wpa_passphrase=wouldn'tyouliketoknow
auth_algs=1
wpa=2
wpa_pairwise=CCMP
ssid=WLAN5
bridge=br-lan
wpa_disable_eapol_key_retries=0
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=no:ta:re:al:ma:c


bss=wlan0-1
ctrl_interface=/var/run/hostapd
ap_isolate=1
ap_max_inactivity=3600
bss_load_update_period=60
chan_util_avg_period=600
disassoc_low_ack=0
preamble=1
wmm_enabled=1
ignore_broadcast_ssid=0
uapsd_advertisement_enabled=1
utf8_ssid=1
wpa_group_rekey=86400
wpa_passphrase=noonecares
auth_algs=1
wpa=2
wpa_pairwise=CCMP
ssid=guest5
bridge=br-guestWLAN
wpa_disable_eapol_key_retries=0
wpa_key_mgmt=WPA-PSK
okc=0
disable_pmksa_caching=1
bssid=no:ta:re:al:ma:c

[greearb]

I think I fixed two issues with the new firmware, both of which could cause crashes. Please see if the attached firmware works better, and if you still see crashes, either post them here or under a new bug as seems appropriate.
It looks like both of your NICs are 9980, so you shouldn't need the 988x firmware, but aside from taking a bit of space, it should not hurt anything.

firmware-5-full-htt-mgt-community.bin.gz

[ghost]

Thankyou. So far no issues. I'll test for a day or two before building openwrt again. I'll test for a few days after that and, assuming no issues, report in a week or so.

root@OpenWrt:~# ethtool -i wlan0 | grep firm
firmware-version: 10.4b-ct-9980-fH-012-648296d

root@OpenWrt:~# ethtool -i wlan1 | grep firm
firmware-version: 10.4b-ct-9980-fH-012-648296d

root@OpenWrt:~# uptime
 10:12:16 up 39 min,  load average: 0.02, 0.06, 0.07

root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy1/ath10k/fw_reset_stats 
fw_crash_counter		0
fw_warm_reset_counter		0
fw_cold_reset_counter		4

root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy0/ath10k/fw_reset_stats 
fw_crash_counter		0
fw_warm_reset_counter		0
fw_cold_reset_counter		4

[greearb]

Thank you for the followup and for the initial big detailed post. I think the gtk rekey disconnect is probably not an ath10k problem since I see it work fine with our ath10k related stations, but another USB realtek NIC disconnects often. Probably this is a station bug, and based on postings from the linux wireless list some time back, probably it is somewhat expected. I am closing this bug since I think I fixed the crashes. Please (re)open individual bugs to track each issue you see in the future. It is a lot easier for me to deal with single-issue bugs. Debugging something like an iphone drop would require a good packet capture and potentially some verbose log capture from the OpenWRT system. If you can provide the capture, I'll be happy to take a look at it.