Netgear R7800 - ath10k QCA 9984 firmware crash

[Brouno]

Description of the problem (how to configure, how to reproduce, how often it happens).

Just flashing the r7800 with the openwrt 18.06.1 factory image (here :
https://downloads.openwrt.org/releases/18.06.1/targets/ipq806x/generic/openwrt-18.06.1-ipq806x-netgear_r7800-squashfs-factory.img)
Then replace /lib/firmware/ath10k/QCA9984/hw1.0/firmware-5.bin (which crash also) with this one:
https://www.candelatech.com/downloads/ath10k-9984-10-4/firmware-5-ct-full-community.bin
(same with other, from kvalo github or stock openwrt 17.01)

I was not able to enable debug as explain here https://www.candelatech.com/ath10k-bugs.php with debug_mask, nor through the modprobe.conf file
No firmware crash file generated

I have to mention that I get a beta hardware of the R7800, perhaps there is something different in my hardware compared to the released one.

Software (OS, Firmware version, kernel, driver, etc)
OpenWRT 18.06.1 on R7800 (https://downloads.openwrt.org/releases/18.06.1/targets/ipq806x/generic/openwrt-18.06.1-ipq806x-netgear_r7800-squashfs-factory.img)

Hardware (NIC chipset, platform, etc)
Netgear R7800

Logs (dmesg, maybe supplicant and/or hostap)
Dmesg :
[ 5652.231181] ath10k_pci 0001:01:00.0: disabling bus mastering
[ 5652.271022] ath10k_pci 0000:01:00.0: disabling bus mastering
[ 5661.414430] ath10k_pci 0000:01:00.0: assign IRQ: got 67
[ 5661.414922] ath10k_pci 0000:01:00.0: enabling bus mastering
[ 5661.415373] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[ 5661.582942] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[ 5661.582999] ath10k_pci 0000:01:00.0: Falling back to user helper
[ 5661.614368] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[ 5661.615259] ath10k_pci 0000:01:00.0: Unknown FW IE: 30
[ 5661.622294] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[ 5661.627345] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[ 5661.639258] ath10k_pci 0000:01:00.0: firmware ver 10.4-ct-9984-fW-010-868495e api 5 features peer-flow-ctrl crc32 af726ff3
[ 5663.109377] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:31 crc32 dd6d039c
[ 5667.233480] ath10k_pci 0000:01:00.0: firmware crashed! (guid cf595b21-45d8-4665-950d-ddaf88d7a8f5)
[ 5667.233559] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[ 5667.241368] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[ 5667.256268] ath10k_pci 0000:01:00.0: firmware ver 10.4-ct-9984-fW-010-868495e api 5 features peer-flow-ctrl crc32 af726ff3
[ 5667.260613] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:31 crc32 dd6d039c
[ 5667.270450] ath10k_pci 0000:01:00.0: htt-ver 0.0 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[ 5667.280118] ath10k_pci 0000:01:00.0: firmware register dump:
[ 5667.287797] ath10k_pci 0000:01:00.0: [00]: 0x01000000 0x000015B3 0x009CBD16 0x00975B31
[ 5667.293703] ath10k_pci 0000:01:00.0: [04]: 0x009CBD16 0x00060530 0x00000011 0x004188C0
[ 5667.301428] ath10k_pci 0000:01:00.0: [08]: 0x00417628 0x00400000 0x004069B0 0x00423080
[ 5667.309326] ath10k_pci 0000:01:00.0: [12]: 0x00000009 0xFFFFFFFF 0x00973B25 0x00973B47
[ 5667.317226] ath10k_pci 0000:01:00.0: [16]: 0x009CBD16 0x000CAFD8 0x00000000 0x00000000
[ 5667.325126] ath10k_pci 0000:01:00.0: [20]: 0x409CBD16 0x004068EC 0x00000001 0x00403628
[ 5667.333025] ath10k_pci 0000:01:00.0: [24]: 0x8098050C 0x0040694C 0x0000003C 0xC09CBD16
[ 5667.340924] ath10k_pci 0000:01:00.0: [28]: 0x800DC4E5 0x004069AC 0x00000004 0x00414D90
[ 5667.348824] ath10k_pci 0000:01:00.0: [32]: 0x800C383A 0x004069DC 0x0002D955 0x00400000
[ 5667.356722] ath10k_pci 0000:01:00.0: [36]: 0x809800A3 0x00406A0C 0x00414D10 0x009C04F4
[ 5667.364622] ath10k_pci 0000:01:00.0: [40]: 0x8098003A 0x00406A5C 0x00400000 0x00400600
[ 5667.372520] ath10k_pci 0000:01:00.0: [44]: 0x809607C1 0x00406A8C 0xD8400100 0x00406A9C
[ 5667.380420] ath10k_pci 0000:01:00.0: [48]: 0x80960892 0x00406AAC 0x00000000 0x00000008
[ 5667.388321] ath10k_pci 0000:01:00.0: [52]: 0x80960E39 0x00406ACC 0x00400000 0x00400600
[ 5667.396219] ath10k_pci 0000:01:00.0: [56]: 0x80960E9D 0x00406AEC 0x00000000 0x00400600
[ 5667.404116] ath10k_pci 0000:01:00.0: Copy Engine register dump:
[ 5667.412021] ath10k_pci 0000:01:00.0: [00]: 0x0004a000 6 6 7 6
[ 5667.417837] ath10k_pci 0000:01:00.0: [01]: 0x0004a400 11 11 10 11
[ 5667.424434] ath10k_pci 0000:01:00.0: [02]: 0x0004a800 0 0 63 0
[ 5667.430857] ath10k_pci 0000:01:00.0: [03]: 0x0004ac00 0 0 0 0
[ 5667.437279] ath10k_pci 0000:01:00.0: [04]: 0x0004b000 0 0 0 0
[ 5667.443704] ath10k_pci 0000:01:00.0: [05]: 0x0004b400 0 0 0 0
[ 5667.450127] ath10k_pci 0000:01:00.0: [06]: 0x0004b800 0 0 0 0
[ 5667.456550] ath10k_pci 0000:01:00.0: [07]: 0x0004bc00 1 1 1 1
[ 5667.462974] ath10k_pci 0000:01:00.0: [08]: 0x0004c000 0 0 127 0
[ 5667.469398] ath10k_pci 0000:01:00.0: [09]: 0x0004c400 1 1 1 1
[ 5667.475821] ath10k_pci 0000:01:00.0: [10]: 0x0004c800 0 0 0 0
[ 5667.482245] ath10k_pci 0000:01:00.0: [11]: 0x0004cc00 0 0 0 0
[ 5668.242481] ath10k_pci 0000:01:00.0: failed to receive control response completion, polling..
[ 5669.273743] ath10k_pci 0000:01:00.0: ctl_resp never came in (-110)
[ 5669.273800] ath10k_pci 0000:01:00.0: failed to connect to HTC: -110
[ 5669.354740] ath10k_pci 0000:01:00.0: could not init core (-110)
[ 5669.355197] ath10k_pci 0000:01:00.0: could not probe fw (-110)
[ 5669.360000] ath10k_pci 0001:01:00.0: assign IRQ: got 100
[ 5669.361506] ath10k_pci 0001:01:00.0: enabling bus mastering
[ 5669.362147] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[ 5669.390530] ath10k_pci 0000:01:00.0: cannot restart a device that hasn't been started
[ 5669.533006] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[ 5669.533052] ath10k_pci 0001:01:00.0: Falling back to user helper
[ 5669.562803] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[ 5669.563641] ath10k_pci 0001:01:00.0: Unknown FW IE: 30
[ 5669.570741] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[ 5669.575777] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[ 5669.587690] ath10k_pci 0001:01:00.0: firmware ver 10.4-ct-9984-fW-010-868495e api 5 features peer-flow-ctrl crc32 af726ff3
[ 5671.058557] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:31 crc32 dd6d039c
[ 5675.188693] ath10k_pci 0001:01:00.0: firmware crashed! (guid 992e358f-d169-42c7-afc2-6b0423bf066a)
[ 5675.188781] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[ 5675.196661] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[ 5675.211645] ath10k_pci 0001:01:00.0: firmware ver 10.4-ct-9984-fW-010-868495e api 5 features peer-flow-ctrl crc32 af726ff3
[ 5675.215829] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:31 crc32 dd6d039c
[ 5675.225665] ath10k_pci 0001:01:00.0: htt-ver 0.0 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[ 5675.235250] ath10k_pci 0001:01:00.0: firmware register dump:
[ 5675.243009] ath10k_pci 0001:01:00.0: [00]: 0x01000000 0x000015B3 0x009CBD16 0x00975B31
[ 5675.248915] ath10k_pci 0001:01:00.0: [04]: 0x009CBD16 0x00060530 0x00000011 0x004188C0
[ 5675.256641] ath10k_pci 0001:01:00.0: [08]: 0x00417628 0x00400000 0x004069B0 0x00423080
[ 5675.264542] ath10k_pci 0001:01:00.0: [12]: 0x00000009 0xFFFFFFFF 0x00973B25 0x00973B47
[ 5675.272441] ath10k_pci 0001:01:00.0: [16]: 0x009CBD16 0x000CAFD3 0x00000000 0x00000000
[ 5675.280339] ath10k_pci 0001:01:00.0: [20]: 0x409CBD16 0x004068EC 0x00000001 0x00403628
[ 5675.288239] ath10k_pci 0001:01:00.0: [24]: 0x8098050C 0x0040694C 0x0000003C 0xC09CBD16
[ 5675.296138] ath10k_pci 0001:01:00.0: [28]: 0x800DC4E5 0x004069AC 0x00000004 0x00414D90
[ 5675.304037] ath10k_pci 0001:01:00.0: [32]: 0x800C383A 0x004069DC 0x0002DA07 0x00400000
[ 5675.311936] ath10k_pci 0001:01:00.0: [36]: 0x809800A3 0x00406A0C 0x00414D10 0x009C04F4
[ 5675.319835] ath10k_pci 0001:01:00.0: [40]: 0x8098003A 0x00406A5C 0x00400000 0x00400600
[ 5675.327734] ath10k_pci 0001:01:00.0: [44]: 0x809607C1 0x00406A8C 0xD8400100 0x00406A9C
[ 5675.335634] ath10k_pci 0001:01:00.0: [48]: 0x80960892 0x00406AAC 0x00000000 0x00000008
[ 5675.343534] ath10k_pci 0001:01:00.0: [52]: 0x80960E39 0x00406ACC 0x00400000 0x00400600
[ 5675.351433] ath10k_pci 0001:01:00.0: [56]: 0x80960E9D 0x00406AEC 0x00000000 0x00400600
[ 5675.359331] ath10k_pci 0001:01:00.0: Copy Engine register dump:
[ 5675.367237] ath10k_pci 0001:01:00.0: [00]: 0x0004a000 6 6 7 6
[ 5675.373053] ath10k_pci 0001:01:00.0: [01]: 0x0004a400 11 11 10 11
[ 5675.379648] ath10k_pci 0001:01:00.0: [02]: 0x0004a800 0 0 63 0
[ 5675.386072] ath10k_pci 0001:01:00.0: [03]: 0x0004ac00 0 0 0 0
[ 5675.392497] ath10k_pci 0001:01:00.0: [04]: 0x0004b000 0 0 0 0
[ 5675.398920] ath10k_pci 0001:01:00.0: [05]: 0x0004b400 0 0 0 0
[ 5675.405342] ath10k_pci 0001:01:00.0: [06]: 0x0004b800 0 0 0 0
[ 5675.411765] ath10k_pci 0001:01:00.0: [07]: 0x0004bc00 1 1 1 1
[ 5675.418190] ath10k_pci 0001:01:00.0: [08]: 0x0004c000 0 0 127 0
[ 5675.424614] ath10k_pci 0001:01:00.0: [09]: 0x0004c400 1 1 1 1
[ 5675.431037] ath10k_pci 0001:01:00.0: [10]: 0x0004c800 0 0 0 0
[ 5675.437461] ath10k_pci 0001:01:00.0: [11]: 0x0004cc00 0 0 0 0
[ 5676.234186] ath10k_pci 0001:01:00.0: failed to receive control response completion, polling..
[ 5677.282438] ath10k_pci 0001:01:00.0: ctl_resp never came in (-110)
[ 5677.282487] ath10k_pci 0001:01:00.0: failed to connect to HTC: -110
[ 5677.364677] ath10k_pci 0001:01:00.0: could not init core (-110)
[ 5677.365149] ath10k_pci 0001:01:00.0: could not probe fw (-110)
[ 5677.410525] ath10k_pci 0001:01:00.0: cannot restart a device that hasn't been started

[greearb]

This is likely calibration data related assert. It would take a bit of time to verify exactly what is failing. Maybe you need a custom board-data file for this platform and you are using some generic board data file instead?

[Brouno]

Yes I think too it can be this kind of issue.
Tell me if you need informations.

The device is currently under factory fw, and I have telnet access to the netgear firmware

I can put back openwrt at any time.

[greearb]

I do not have time to work on this. It is more of a platform issue than a firmware issue. Closing this, hopefully someone in OpenWRT or similar community can help you get a valid board data.

[greearb]

On 2/21/19 12:40 PM, Brouno wrote: Hi, I know this one is closed, but I made some progress. And I just request some advices if it's possible Using last openwrt snapshot, no more firmware crash, but I get the following error now : [ 21.916566] ath: EEPROM regdomain: 0xffff [ 21.916581] ath: EEPROM indicates we should expect a country code [ 21.916599] ath: invalid regulatory domain/country code 0xbfff [ 21.916609] ath: Invalid EEPROM contents Look like a regulatory issue probably related to beta hardware. Is there a possibility to correct that ? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAR2quR-6aFd_LU7y2t-auzCxXYTfNgFks5vPwRQgaJpZM4Ww9I->.

My ath10k-ct driver allows you to over-ride the regulatory domain using the fwcfg file or module options. https://www.candelatech.com/ath10k-10.4.php#config You *really* should not override the regulatory domain unless you understand the implications and possible violation of FCC or similar procedures. Thanks, Ben

…

-- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com

[Brouno]

Hi Ben,

Thanks a lot, you rocks !! :-) 🥇

It's now working, putting the correct regdom value (french), here is the dmseg log 👍

[ 24.577950] ath10k_pci 0001:01:00.0: DANGER! You're overriding EEPROM-defined regulatory domain
[ 24.577987] ath10k_pci 0001:01:00.0: from: 0x0 to 0xfa (svc-ready-work)
[ 24.585544] ath10k_pci 0001:01:00.0: Your card was not certified to operate in the domain you chose.
[ 24.592053] ath10k_pci 0001:01:00.0: This might result in a violation of your local regulatory rules.
[ 24.601508] ath10k_pci 0001:01:00.0: Do not ever do this unless you really know what you are doing!
[ 24.611654] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96
[ 24.619552] ath10k_pci 0001:01:00.0: msdu-desc: 2500 skid: 32
[ 24.705337] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0'
[ 24.706236] ath10k_pci 0001:01:00.0: wmi print 'free: 85544 iram: 26772 sram: 18240'
[ 24.968483] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[ 25.066318] ath: EEPROM regdomain: 0x80fa
[ 25.066332] ath: EEPROM indicates we should expect a country code
[ 25.066345] ath: doing EEPROM country->regdmn map search
[ 25.066358] ath: country maps to regdmn code: 0x37
[ 25.066372] ath: Country alpha2 being used: FR
[ 25.066383] ath: Regpair used: 0x37

If we meet one day consider I owe you a beer ;-) !