Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wlan freezes in raspberry pi 3/PiZeroW (Not 3B+) #1342

Open
dh-connect opened this issue Mar 11, 2016 · 477 comments
Open

wlan freezes in raspberry pi 3/PiZeroW (Not 3B+) #1342

dh-connect opened this issue Mar 11, 2016 · 477 comments

Comments

@dh-connect
Copy link

@dh-connect dh-connect commented Mar 11, 2016

I put the same sd card (running debian 8 jessie, kernel 4.1.19) from the raspberry pi 2 with usb wifi (EDIMAX EW-7811UN Wireless USB Adapter, 150 Mbit/s, IEEE802.11b/g/n) into the new raspberry pi 3 using integrated wlan. Since then the wlan freezes after while (several hours) of usage couldn't find out if it's due to havy wifi usage or not, because I haven't change the software I guess it has to do with the new hardware. When the wlan freezes the pi can't be reached any longer, neither ifdown + ifup nor restart networking service helps in this case, I have to reboot the system to get it back to work, syslog doesn't say much only this:
dhcpcd[522]: wlan0: fe80::8af7:c7ff:fece:5912: expired option 25,

I've tried to change these settings so far, but without improvement:

sudo nano /etc/network/interfaces
wireless-power off

sudo nano /etc/sysctl.conf
at the end of the file add the following line
vm.min_free_kbytes = 16384

sudo nano /boot/cmdline.txt
At the end of the line, add:
smsc95xx.turbo_mode=N
dwc_otg.dma_enable=1 dwc_otg.dma_burst_size=256

@clivem
Copy link
Contributor

@clivem clivem commented Mar 11, 2016

EDIMAX EW-7811UN.... That's using rtl8188cus chipset, IIRC.

If you haven't already got one, create /etc/modprobe.d/8192cu.conf, with content....

Disable power management

options 8192cu rtw_power_mgnt=0 rtw_enusbss=0

@grbd
Copy link

@grbd grbd commented Mar 11, 2016

The rpi3 actually uses the brcmfmac driver for the inbuilt wifi
there is an issue that requires the power saving / mangement to be turned off

I think the newer raspian kernels have patched this already to disable power saving by default but I don't think it's in this 4.5 branch yet

What I'm doing at the moment (gentoo install) is the following at bootup to disable the power saving on the wifi card

iw wlan0 set power_save off

@clivem
Copy link
Contributor

@clivem clivem commented Mar 11, 2016

The rpi3 actually uses the brcmfmac driver for the inbuilt wifi

Yes, I know. Oh I see. He's not using the EDIMAX EW-7811UN dongle anymore. He used to use it with RPi2.

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 11, 2016

yes I don't use the usb wifi any longer, how do I set up the cmd line to turn off the power management?
crontab
@reboot iw wlan0 set power_save off

@grbd
Copy link

@grbd grbd commented Mar 11, 2016

Not sure for raspian, since I'm using gentoo it'll be different

@pelwell
Copy link
Contributor

@pelwell pelwell commented Mar 11, 2016

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 14, 2016

Seems to work since I have turned the powermanagement off I haven't had another wlan crash.

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 14, 2016

Just to mentioned it, to restart the wlan automatically after a crash, this here helps:
sudo cp /etc/wpa_supplicant/ifupdown.sh /etc/ifplugd/action.d/ifupdown

@popcornmix
Copy link
Collaborator

@popcornmix popcornmix commented Mar 14, 2016

BTW, latest apt-get upgrade kernel has power management disabled by default.
@dh-connect does this work for you if you remove your current workaround?

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 21, 2016

it's still crashing after the latest upgrade, now i get this error in syslog:
brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!

@pelwell
Copy link
Contributor

@pelwell pelwell commented Mar 21, 2016

When you say it's crashing, are there symptoms other than the error message?

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 21, 2016

no, just the one I have posted here but it is in the log many times

the wlan stops working, i can still work with it but to get the wlan back working I have to reboot it

@pelwell
Copy link
Contributor

@pelwell pelwell commented Mar 21, 2016

Thanks - I think "wlan stops working" counts as a symptom.

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 21, 2016

I've tried a few things, but wlan still breaks down

to answer the question above when I take back the configuration
wireless-power off in /etc/network/interfaces
and reboot
and check the settings with iwconfig
the power management ist turned back on so by default I wouldn't say that this is diasbled so I will leave the configuration

I tried that with kernel 4.1.19 and now also with kernel 4.1.20 ... no change

when the wlan crashed and i try to turn it back on with ifdown and ifup wlan0 I get this:
Error for wireless request "Set Power Management" (8B2C) : SET failed on device wlan0 ; Invalid exchange.

I also got a few more error in syslog:

dhcpcd[532]: wlan0: xxx: expired option 25

Mar 21 17:29:35 raspberrypi kernel: [ 6627.337503] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52
Mar 21 17:29:36 raspberrypi wpa_supplicant[6318]: Successfully initialized wpa_supplicant
Mar 21 17:29:36 raspberrypi dhcpcd[532]: wlan0: carrier lost

Mar 21 17:29:43 raspberrypi kernel: [ 6635.337616] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52

Mar 21 17:29:45 raspberrypi kernel: [ 6637.337588] brcmfmac: brcmf_do_escan: error (-52)
Mar 21 17:29:45 raspberrypi kernel: [ 6637.337602] brcmfmac: brcmf_cfg80211_scan: scan error (-52)

Mar 21 17:29:47 raspberrypi kernel: [ 6639.337596] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -52
Mar 21 17:29:49 raspberrypi kernel: [ 6641.337632] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -52

is there anything else I could try?

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 22, 2016

also these:

Mar 21 21:26:55 raspberrypi dhcpcd[526]: wlan0: xxx: expired option 25
Mar 21 21:28:54 raspberrypi kernel: [ 1958.899715] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
Mar 21 21:30:16 raspberrypi dhcpcd[526]: wlan0: xxx is unreachable, expiring it

@pelwell
Copy link
Contributor

@pelwell pelwell commented Mar 22, 2016

I'm not surprised that iwconfig thinks the device has power-saving enabled - I blocked it within the driver itself, and either the state is saved in the higher layers or there is another change required in order to report it correctly. Either way, the evidence is strong that we have avoided the power-saving bugs, but some other problems still remain.

Do you have any rough figures for the time-to-failure and roughly how much data might have been transferred (from ifconfig)?

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 22, 2016

yes I do, when I have just the webserver running with not much traffic (less than 100 MB) it lasts a day or two, when i transfer large data files like 1 GB wlan crashes within 1 hour

@dh-connect
Copy link
Author

@dh-connect dh-connect commented Mar 29, 2016

anything I can provide to help to find the bug?

here are some error from syslog:

Mar 29 14:20:56 raspberrypi dhcpcd[535]: wlan0: xxx: expired option 25
Mar 29 14:30:15 raspberrypi dhcpcd[535]: wlan0: xxx is unreachable, expiring it
Mar 29 17:18:42 raspberrypi kernel: [186148.102420] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
Mar 29 17:18:43 raspberrypi kernel: [186149.101045] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
Mar 29 17:18:43 raspberrypi kernel: [186149.101145] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
Mar 29 17:18:44 raspberrypi kernel: [186150.101209] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
Mar 29 17:18:50 raspberrypi wpa_supplicant[478]: wlan0: CTRL-EVENT-DISCONNECTED bssid=xxx reason=3 locally_generated=1
Mar 29 17:18:50 raspberrypi kernel: [186156.181033] brcmfmac: brcmf_cfg80211_disconnect: error (-52)
Mar 29 17:18:52 raspberrypi kernel: [186158.181028] brcmfmac: send_key_to_dongle: wsec_key error (-52)
Mar 29 17:18:54 raspberrypi kernel: [186160.181046] brcmfmac: send_key_to_dongle: wsec_key error (-52)
Mar 29 17:18:56 raspberrypi kernel: [186162.181048] brcmfmac: send_key_to_dongle: wsec_key error (-52)
Mar 29 17:18:58 raspberrypi kernel: [186164.181049] brcmfmac: send_key_to_dongle: wsec_key error (-52)
Mar 29 17:18:58 raspberrypi kernel: [186164.185477] cfg80211: Calling CRDA to update world regulatory domain
Mar 29 17:18:58 raspberrypi dhcpcd[535]: wlan0: carrier lost
Mar 29 17:18:58 raspberrypi wpa_supplicant[7354]: Successfully initialized wpa_supplicant
Mar 29 17:18:58 raspberrypi kernel: [186164.314511] brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code
Mar 29 17:18:58 raspberrypi kernel: [186164.314541] cfg80211: World regulatory domain updated:
Mar 29 17:18:58 raspberrypi kernel: [186164.314548] cfg80211: DFS Master region: unset
Mar 29 17:18:58 raspberrypi kernel: [186164.314555] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)
Mar 29 17:18:58 raspberrypi kernel: [186164.314565] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Mar 29 17:18:58 raspberrypi kernel: [186164.314573] cfg80211: (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)
Mar 29 17:18:58 raspberrypi kernel: [186164.314581] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A)
Mar 29 17:18:58 raspberrypi kernel: [186164.314592] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)
Mar 29 17:18:58 raspberrypi kernel: [186164.314602] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)
Mar 29 17:18:58 raspberrypi kernel: [186164.314611] cfg80211: (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s)
Mar 29 17:18:58 raspberrypi kernel: [186164.314645] cfg80211: (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A)
Mar 29 17:18:58 raspberrypi kernel: [186164.314654] cfg80211: (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)

@pelwell
Copy link
Contributor

@pelwell pelwell commented Mar 30, 2016

Thanks for the offer, but this is in the hands of Broadcom now.

@juched78
Copy link

@juched78 juched78 commented May 5, 2016

Any update from Broadcom if this is a bug which will be fixed? I now have a cron job setup to bring down and up wlan0 when it fails to ping.

@dh-connect
Copy link
Author

@dh-connect dh-connect commented May 5, 2016

quick update from my side, i could get the problem fixed seems to be driver related, i installed Ubuntu MATE 16.04 with kernel 4.4.8 and haven't had any problems with wifi since

i mean they advertise is: "Ubuntu MATE 16.04 also has fully working Bluetooth and Wifi on the Raspberry Pi 3" which seems true

maybe it also works with a new Debian release, which i can not tell

@pelwell
Copy link
Contributor

@pelwell pelwell commented May 5, 2016

@juched78 Are you running a 4.4 kernel? If not, please run sudo rpi-update to get the latest 4.4.8 build and see if that suffers the same problem.

The Broadcom drivers have changed significantly since 4.1, and our 4.4 tree includes back-ports of some fixes that went into 4.5. I'm not aware of any outstanding bugs apart from the failure to wake from sleep (power management is still disabled) - channels 12 & 13 are usable where permitted, and Ad Hoc mode doesn't crash - but there may still be lurking issues.

@pelwell
Copy link
Contributor

@pelwell pelwell commented May 5, 2016

Oh, there is one reported bug still in 4.4.8 - apparently heavy use of hostapd can lead to a kernel warning (see #1375).

@juched78
Copy link

@juched78 juched78 commented May 5, 2016

I am running:
Linux XXX 4.4.8-v7+ #880 SMP Fri Apr 22 21:55:04 BST 2016 armv7l GNU/Linux

Apr 27 2016 11:06:18
Copyright (c) 2012 Broadcom
version 9b52ab7b475f4a056658fd2d95d2440b32167390 (clean) (release)

With my Netgear R7000 running Shibby Tomato, around 2 days in the wifi drops, and in the sys logs I see:

CTRL-EVENT-DISCONNECTED
brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-52)
brcmfmac: send_key_to_dongle: wsec_key error (-52)
...
brcmfmac: brcmf_do_escan: error (-52)
...
wpa_supplicant[506]: wlan0: CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
...
brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code

(then I see it scan and re-pick my country code CA)

brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -52
brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -52
brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52

Then it seems to never reconnect...

Using sudo ifdown wlan0 followed by sudo ifup wlan0 brings back my connection.

@juched78
Copy link

@juched78 juched78 commented May 5, 2016

Just upgraded to:
Linux JuchePi 4.4.8-v7+ #881 SMP Sat Apr 30 12:16:50 BST 2016 armv7l GNU/Linux

Not sure what is all different from the 22nd to the 30th. I will monitor the connection.

@maciex
Copy link

@maciex maciex commented May 9, 2016

My RPi 3 also hit that problem. I got few different kernel messages. Mainly one of those below.
After that I can' get the WiFi working, bringing wlan0 down then up does not help.

May 09 21:24:25 osmc kernel: brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
May 09 22:00:15 osmc kernel: brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
May 09 22:00:18 osmc kernel: brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
May 10 00:51:10 osmc kernel: brcmfmac: brcmf_cfg80211_get_tx_power: error (-52)
May 10 00:51:12 osmc kernel: brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
May 10 00:53:16 osmc kernel: brcmfmac: brcmf_do_escan: error (-52)
May 10 00:53:16 osmc kernel: brcmfmac: brcmf_cfg80211_scan: scan error (-52)

Raspberry is powered from original power adapter for version 3. I'm running latest OSMC:
$ uname -a
Linux osmc 4.4.8-3-osmc #1 SMP PREEMPT Sun May 1 18:57:43 UTC 2016 armv7l GNU/Linux

@juched78
Copy link

@juched78 juched78 commented May 11, 2016

Still monitoring. I had openhab go offline after running 3 days but for some reason I could still ssh into the Pi which I usually couldn't. The top of the hour and the wifi script ran to bring down and bring up the connection and then it reconnected to my openhab org. Odd. Will keep watching.

@duncanmcdowell
Copy link

@duncanmcdowell duncanmcdowell commented May 14, 2016

I am also experiencing the same issue - dmesg trace as follows:

send_key_to_dongle: wsec_key error (-52)
brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -52
brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
brcmf_cfg80211_get_tx_power: error (-52)

Usage:

rp3 is being used as a router/access point

Connectivity length seems random - I've had as high as two weeks, and as bad as a few minutes. Lately it's been going out every 20 minutes or so. Bringing wlan0 down and back up does not resolve the issue - a full reboot is required.

Problem seems to be exacerbated while streaming Netflix from my AppleTV. Though this was not the case when I had the two weeks of uptime.

I'm on 4.4.10-v7+

@maciex
Copy link

@maciex maciex commented May 15, 2016

I switched the channel from 13 to 6 to check if that could be the problem (there were some defects about the high channels) and since then I haven't had a WiFi freez. But that could be a coincidence...

@maciex
Copy link

@maciex maciex commented May 16, 2016

Changing access point channels didn't help. WiFi still breaks. Last few times I had to restart few times in a row to get it working.

@Tos26
Copy link

@Tos26 Tos26 commented Oct 3, 2019

No one to answer or escalate this issue?

big-henry pushed a commit to big-henry/mirror_ubuntu-bionic-kernel that referenced this issue Feb 17, 2020
This is to workaround for a possible issue in the
wireless chip firmware where some packets with
higher priorities seem to go missing.

See raspberrypi/linux#1342 for
details.
YadongQi pushed a commit to YadongQi/ubuntu-bionic that referenced this issue Mar 10, 2020
This is to workaround for a possible issue in the
wireless chip firmware where some packets with
higher priorities seem to go missing.

See raspberrypi/linux#1342 for
details.
@kkoenen
Copy link

@kkoenen kkoenen commented Apr 30, 2020

Same issue. Any news on this?

Apr 29 22:47:04 raspberrypi kernel: [37515.093582] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted

Apr 29 22:47:06 raspberrypi kernel: [37517.524316] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout

Apr 29 22:47:06 raspberrypi kernel: [37517.524776] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle

Apr 29 22:47:06 raspberrypi kernel: [37517.524792] brcmfmac: brcmf_run_escan: error (-110)

Apr 29 22:47:06 raspberrypi kernel: [37517.524807] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

Tried turning off power management for now. Is this an old bug reintroduced?

https://patchwork.kernel.org/patch/9948825/

@lategoodbye
Copy link
Contributor

@lategoodbye lategoodbye commented May 1, 2020

Same issue. Any news on this?

This message only say that the firmware of the wifi chip crashed. There is nothing the Linux kernel can do except of reseting. A helpful bug report contains the following information:

Which wifi firmware you are using?
How do you operate the wifi (AP, client, ...)?
Can you reproduce this within a defined time?
What other wifi devices are involved?

@DEvil0000
Copy link

@DEvil0000 DEvil0000 commented May 1, 2020

@jeremyn54
Copy link

@jeremyn54 jeremyn54 commented May 13, 2020

Same issue. Any news on this?

Apr 29 22:47:04 raspberrypi kernel: [37515.093582] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted

Apr 29 22:47:06 raspberrypi kernel: [37517.524316] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout

Apr 29 22:47:06 raspberrypi kernel: [37517.524776] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle

Apr 29 22:47:06 raspberrypi kernel: [37517.524792] brcmfmac: brcmf_run_escan: error (-110)

Apr 29 22:47:06 raspberrypi kernel: [37517.524807] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

Tried turning off power management for now. Is this an old bug reintroduced?

https://patchwork.kernel.org/patch/9948825/

No solution to speak of, but I have exactly the same problem on a Rpi4 with latest firmware installed. I rolled it back to an SD card image I made a few months ago and the problem is gone. As I'm using hostapd, I believe one of or a combination of these upgrades broke it for me:

$ apt list --upgradeable
Listing... Done
...
hostapd/stable 2:2.7+git20190128+0c1e29f-6+deb10u2 armhf [upgradable from: 2:2.7+git20190128+0c1e29f-6+deb10u1]
firmware-brcm80211/testing 1:20190114-1+rpt6 all [upgradable from: 1:20190114-1+rpt4]
raspberrypi-kernel/testing 1.20200212-1 armhf [upgradable from: 1.20200114-1]
...

I tried turning off power management also (and confirmed it was off with iwconfig), but it had no effect when running hostapd. I will have to forego firmware upgrades until it's fixed, as we are sending out many of these and need stable APs for our customers.

@pelwell
Copy link
Contributor

@pelwell pelwell commented Jun 2, 2020

Anyone experiencing firmware traps, timeouts (-110) etc. - please enable some firmware debugging so we can gather some data.

Add brcmfmac.debug=0x100000 to /boot/cmdline.txt, keeping it in a single, long line, then reboot. Running dmesg | grep brcmfmac should result in output like this:

[    7.650239] brcmfmac: CONSOLE: d 0
[    7.650256] brcmfmac: CONSOLE: 000000.063 wl0: Broadcom BCM4345 802.11 Wireless Controller 7.45.202 (r724630 CY)
[    7.650270] brcmfmac: CONSOLE: 000000.064 TCAM: 256 used: 252 exceed:0
[    7.650284] brcmfmac: CONSOLE: 000000.065 reclaim section 1: Returned 122844 bytes to the heap
[    7.650297] brcmfmac: CONSOLE: 000000.065 reclaim section 4: Returned 44 bytes to the heap
[    7.650310] brcmfmac: CONSOLE: 000000.065 sdpcmd_dpc: Enable
...

Then just carry on as normal. When the brcmfmac firmware dies, capture the output of dmesg to a file and attach it (or a link to pastebin, etc.) here.

Since the failure triggers other kernel messages, there is a danger that the useful output is lost before you have chance to capture it. A way to avoid this is to leave a shell constantly saving the kernel messages to a file:

$ dmesg -w > kernel_log.txt &
@robgil
Copy link

@robgil robgil commented Jun 12, 2020

Seeing the same issue here. Will try the debug mentioned above.

Running hostapd in AP mode, wireguard, and frr. Also using Sixfab cellular hat.

[46972.803286] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[46975.363309] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[46975.363322] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[47292.885392] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47295.445423] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47295.445436] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[47602.007429] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47604.567452] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47604.567465] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[47830.248947] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47838.328989] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[47887.049300] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110
[47892.649358] brcmfmac: brcmf_cfg80211_stop_ap: SET SSID error (-110)
[47895.209353] brcmfmac: brcmf_cfg80211_stop_ap: BRCMF_C_DOWN error -110
[47897.769374] brcmfmac: brcmf_cfg80211_stop_ap: setting AP mode failed -110
[47902.889420] brcmfmac: brcmf_cfg80211_stop_ap: BRCMF_C_UP error -110
[47905.449430] brcmfmac: brcmf_set_mpc: fail to set mpc
Linux raspberrypi 4.19.118-v7+ #1311 SMP Mon Apr 27 14:21:24 BST 2020 armv7l GNU/Linux
@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Jul 13, 2020

I am also able to recreate this on the 5.4 branch. FWIW, I can always manually trigger this bug by SCPing a large file (>400MB) to my Pi Zero W.

If it helps, my kernel version is as of this commit - 3c860a6

# uname -a
Linux pichime-1-c93bb27a 5.4.50 #1 Sun Jul 12 20:53:57 CDT 2020 armv6l GNU/Linux
# dmesg | grep brcmfmac | grep Firmware
[    5.319134] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: May  2 2019 02:39:18 version 7.45.98.83 (r714225 CY) FWID 01-e539531f

Crash Log with Debugging:

[  340.321646] ieee80211 phy1: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  342.881642] ieee80211 phy1: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  345.441616] ieee80211 phy1: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  348.001649] ieee80211 phy1: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  358.241623] ieee80211 phy1: brcmf_cfg80211_disconnect: error (-110)
[  363.361640] ieee80211 phy1: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  371.041641] ieee80211 phy1: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  373.601642] ieee80211 phy1: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  376.161620] ieee80211 phy1: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  376.170775] ieee80211 phy1: brcmf_cfg80211_reg_notifier: Country code iovar returned err = -110
[  383.841632] ieee80211 phy1: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  383.851056] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[  386.401643] ieee80211 phy1: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  388.961642] ieee80211 phy1: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  391.521632] ieee80211 phy1: brcmf_cfg80211_set_power_mgmt: error (-110)
[  394.081651] ieee80211 phy1: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  409.521619] ieee80211 phy1: brcmf_run_escan: error (-110)
[  409.527146] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)
[  412.081641] ieee80211 phy1: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  414.641643] ieee80211 phy1: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  417.201652] ieee80211 phy1: brcmf_run_escan: error (-110)
[  417.207175] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)
[  419.761655] ieee80211 phy1: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  424.881645] ieee80211 phy1: brcmf_run_escan: error (-110)
[  424.887168] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)
[  430.001645] ieee80211 phy1: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  432.561651] ieee80211 phy1: brcmf_run_escan: error (-110)
[  432.567172] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)
[  435.121637] ieee80211 phy1: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  437.681648] ieee80211 phy1: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  440.241651] ieee80211 phy1: brcmf_run_escan: error (-110)
[  440.247173] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)
[  447.921623] ieee80211 phy1: brcmf_run_escan: error (-110)
[  447.927145] ieee80211 phy1: brcmf_cfg80211_scan: scan error (-110)

During the above crash I ran an ifdown and ifup which didn't restore wifi. Only solution is to either reboot the pi, or rmmod & modprobe brcmfmac.

It's worth noting this happens with power management turned off, since I have this in my interfaces file:

pre-up iwconfig wlan0 power off
@pelwell
Copy link
Contributor

@pelwell pelwell commented Jul 13, 2020

That isn't the most recent firmware for the 43438 - we're now on:

Version: 7.45.98.94 (r723000 CY) CRC: ba33fa65 Date: Tue 2019-10-22 02:01:06 PDT Ucode Ver: 1043.2137 FWID 01-3b33decd

Try updating your firmware-brcm80211 package, or downloading the firmware from: https://github.com/RPi-Distro/firmware-nonfree/

If you still see errors, enable brcmfmac firmware logging by adding brcmfmac.debug=0x100000 to cmdline.txt.

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Jul 14, 2020

@pelwell Sorry about that, but I updated and can still recreate the issue using the method I mentioned.

Note I enabled debugging as requested, but this is all I got:

[    0.000000] Kernel command line: root=/dev/mmcblk0p2 8250.nr_uarts=1 console=ttyS0,115200 rootwait earlyprintk brcmfmac.debug=0x100000
[    4.940560] brcmfmac: F1 signature read @0x18000000=0x1541a9a6
[    4.958767] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[    4.973290] usbcore: registered new interface driver brcmfmac
[    5.324551] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43430-sdio for chip BCM43430/1
[    5.334223] brcmfmac: brcmf_c_process_clm_blob: no clm_blob available (err=-2), device may have limited channels available
[    5.347276] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: Oct 22 2019 01:59:28 version 7.45.98.94 (r723000 CY) FWID 01-3b33decd
[    5.443617] brcmfmac: CONSOLE: hndarm_armr addr: 0x18003000, cr4_idx: 0
[    5.443635] brcmfmac: CONSOLE: 000000.001
[    5.443646] brcmfmac: CONSOLE: RTE (SDIO-CDC) 7.45.98.94 (r723000 CY) on BCM43430 r1 @ 37.4/81.6/81.6MHz
[    5.443655] brcmfmac: CONSOLE: 000000.003 sdpcmdcdc0: Broadcom SDPCMD CDC driver
[    5.443665] brcmfmac: CONSOLE: 000000.008 reclaim section 0: Returned 46092 bytes to the heap
[    5.443673] brcmfmac: CONSOLE: 000000.012 wlc_bmac_info_init: host_enab 1
[    5.443684] brcmfmac: CONSOLE: 000000.064 wl0: Broadcom BCM43430 802.11 Wireless Controller 7.45.98.94 (r723000 CY)
[    5.443693] brcmfmac: CONSOLE: 000000.067 TCAM: 256 used: 212 exceed:0
[    5.443702] brcmfmac: CONSOLE: 000000.069 reclaim section 1: Returned 81228 bytes to the heap
[   51.183451] brcmfmac: CONSOLE: 000045.943 wl0: wl_open
[   51.213694] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[  260.001321] ieee80211 phy0: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  262.561331] ieee80211 phy0: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  265.121296] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  267.681321] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  275.361321] ieee80211 phy0: brcmf_cfg80211_disconnect: error (-110)
[  280.481324] ieee80211 phy0: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  285.601297] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  285.610456] ieee80211 phy0: brcmf_cfg80211_reg_notifier: Country code iovar returned err = -110
[  288.161325] ieee80211 phy0: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  290.721325] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  293.281314] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[  293.291034] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[  300.961315] ieee80211 phy0: brcmf_cfg80211_set_power_mgmt: error (-110)
[  306.081321] ieee80211 phy0: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  308.641320] ieee80211 phy0: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  313.761330] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  324.001323] ieee80211 phy0: brcmf_run_escan: error (-110)
[  324.006845] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[  326.561329] ieee80211 phy0: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  329.121322] ieee80211 phy0: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  331.681324] ieee80211 phy0: brcmf_run_escan: error (-110)
[  331.686848] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[  334.241329] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  339.361315] ieee80211 phy0: brcmf_run_escan: error (-110)
[  339.366836] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[  344.481323] ieee80211 phy0: _brcmf_set_multicast_list: Setting mcast_list failed, -110
[  347.041339] ieee80211 phy0: brcmf_run_escan: error (-110)
[  347.046862] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[  349.601345] ieee80211 phy0: _brcmf_set_multicast_list: Setting allmulti failed, -110
[  352.161310] ieee80211 phy0: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
[  354.721371] ieee80211 phy0: brcmf_run_escan: error (-110)
[  354.726896] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[  362.401325] ieee80211 phy0: brcmf_run_escan: error (-110)
[  362.406850] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Jul 17, 2020

I was able to get more of a log by doing an ifdown & ifup on wlan0, hopefully this helps somewhat:

[ 1420.259650] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[ 1423.774141] ieee80211 phy0: brcmf_run_escan: error (-110)
[ 1423.779662] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[ 1427.294190] ieee80211 phy0: brcmf_run_escan: error (-110)
[ 1427.299710] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[ 1430.814146] ieee80211 phy0: brcmf_run_escan: error (-110)
[ 1430.819668] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-110)
[ 1444.148281] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 1445.157155] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 1446.166847] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 1447.176537] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 1448.185305] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)

...
ifdown and ifup
...

[ 2984.008316] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
[ 2984.019327] ieee80211 phy0: brcmf_run_escan: error (-52)
[ 2984.024840] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
[ 3005.603730] ieee80211 phy0: brcmf_run_escan: error (-52)
[ 3005.609162] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
[ 3005.620132] ieee80211 phy0: brcmf_run_escan: error (-52)
[ 3005.625685] ieee80211 phy0: brcmf_cfg80211_scan: scan error (-52)
[ 3349.033428] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 3349.040692] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 3349.324019] ------------[ cut here ]------------
[ 3349.330137] WARNING: CPU: 0 PID: 262 at net/wireless/sme.c:756 __cfg80211_connect_result+0x41c/0x4d0 [cfg80211]
[ 3349.340546] Modules linked in: ipv6 nf_defrag_ipv6 brcmfmac brcmutil sha256_generic libsha256 cfg80211 rfkill snd_soc_simple_card snd_soc_simple_card_utils snd_soc_max98357a snd_soc_bcm2835_i2s regmap_mmio snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd
[ 3349.365074] CPU: 0 PID: 262 Comm: kworker/u2:2 Not tainted 5.4.51 #1
[ 3349.371533] Hardware name: BCM2835
[ 3349.376401] Workqueue: cfg80211 cfg80211_event_work [cfg80211]
[ 3349.382516] Backtrace:
[ 3349.385049] [<c00156e8>] (dump_backtrace) from [<c0015a34>] (show_stack+0x20/0x24)
[ 3349.392805]  r7:000002f4 r6:bf10d624 r5:00000009 r4:bf135900
[ 3349.398587] [<c0015a14>] (show_stack) from [<c0736d54>] (dump_stack+0x20/0x28)
[ 3349.406004] [<c0736d34>] (dump_stack) from [<c00239a4>] (__warn+0xd0/0x104)
[ 3349.413150] [<c00238d4>] (__warn) from [<c0023d58>] (warn_slowpath_fmt+0x6c/0xc4)
[ 3349.420765]  r7:bf10d624 r6:000002f4 r5:bf135900 r4:00000000
[ 3349.427938] [<c0023cf0>] (warn_slowpath_fmt) from [<bf10d624>] (__cfg80211_connect_result+0x41c/0x4d0 [cfg80211])
[ 3349.438495]  r8:d8dd6084 r7:d94ebe64 r6:00000000 r5:d8dd6004 r4:d8f2da0c
[ 3349.448017] [<bf10d208>] (__cfg80211_connect_result [cfg80211]) from [<bf0dda00>] (cfg80211_process_wdev_events+0x138/0x1c0 [cfg80211])
[ 3349.460512]  r7:d8dd6024 r6:d8dd6004 r5:80000013 r4:d8f2da00
[ 3349.469003] [<bf0dd8c8>] (cfg80211_process_wdev_events [cfg80211]) from [<bf0ddac8>] (cfg80211_process_rdev_events+0x40/0x98 [cfg80211])
[ 3349.481589]  r10:d88bc0d8 r9:00000000 r8:00000000 r7:d948ae00 r6:00000040 r5:d88bc420
[ 3349.489599]  r4:d8dd6004
[ 3349.494901] [<bf0dda88>] (cfg80211_process_rdev_events [cfg80211]) from [<bf0d71c4>] (cfg80211_event_work+0x24/0x2c [cfg80211])
[ 3349.506686]  r5:c772d600 r4:d88bc0d4
[ 3349.511718] [<bf0d71a0>] (cfg80211_event_work [cfg80211]) from [<c003ddd4>] (process_one_work+0x1c8/0x470)
[ 3349.521648]  r5:c772d600 r4:d88bc0d4
[ 3349.525355] [<c003dc0c>] (process_one_work) from [<c003e0c4>] (worker_thread+0x48/0x52c)
[ 3349.533641]  r10:d940d200 r9:00000088 r8:c0a3c760 r7:d940d214 r6:c772d614 r5:d940d200
[ 3349.541603]  r4:c772d600
[ 3349.544279] [<c003e07c>] (worker_thread) from [<c00434cc>] (kthread+0x120/0x15c)
[ 3349.551812]  r10:d0067e88 r9:d8ef3f98 r8:c772d600 r7:d94ea000 r6:00000000 r5:c502c460
[ 3349.559821]  r4:d8ef3f80
[ 3349.562456] [<c00433ac>] (kthread) from [<c00090ac>] (ret_from_fork+0x14/0x28)
[ 3349.569801] Exception stack(0xd94ebfb0 to 0xd94ebff8)
[ 3349.574990] bfa0:                                     00000000 00000000 00000000 00000000
[ 3349.583349] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3349.591665] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 3349.598439]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c00433ac
[ 3349.606436]  r4:c502c460
[ 3349.609020] ---[ end trace 53428b45b18f1d66 ]---
[ 3726.022943] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 3726.030239] ieee80211 phy0: brcmf_cfg80211_scan: Connectinghttps://www.youtube.com/: status (3)
[ 3726.314103] ------------[ cut here ]------------
[ 3726.320236] WARNING: CPU: 0 PID: 262 at net/wireless/sme.c:756 __cfg80211_connect_result+0x41c/0x4d0 [cfg80211]
[ 3726.330648] Modules linked in: ipv6 nf_defrag_ipv6 brcmfmac brcmutil sha256_generic libsha256 cfg80211 rfkill snd_soc_simple_card snd_soc_simple_card_utils snd_soc_max98357a snd_soc_bcm2835_i2s regmap_mmio snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd
[ 3726.355180] CPU: 0 PID: 262 Comm: kworker/u2:2 Tainted: G        W         5.4.51 #1
[ 3726.363093] Hardware name: BCM2835
[ 3726.367928] Workqueue: cfg80211 cfg80211_event_work [cfg80211]
[ 3726.373983] Backtrace:
[ 3726.376518] [<c00156e8>] (dump_backtrace) from [<c0015a34>] (show_stack+0x20/0x24)
[ 3726.384275]  r7:000002f4 r6:bf10d624 r5:00000009 r4:bf135900
[ 3726.390113] [<c0015a14>] (show_stack) from [<c0736d54>] (dump_stack+0x20/0x28)
[ 3726.397538] [<c0736d34>] (dump_stack) from [<c00239a4>] (__warn+0xd0/0x104)
[ 3726.404673] [<c00238d4>] (__warn) from [<c0023d58>] (warn_slowpath_fmt+0x6c/0xc4)
[ 3726.412331]  r7:bf10d624 r6:000002f4 r5:bf135900 r4:00000000
[ 3726.419466] [<c0023cf0>] (warn_slowpath_fmt) from [<bf10d624>] (__cfg80211_connect_result+0x41c/0x4d0 [cfg80211])
[ 3726.430020]  r8:d8dd6084 r7:d94ebe64 r6:00000000 r5:d8dd6004 r4:c5343a0c
[ 3726.439551] [<bf10d208>] (__cfg80211_connect_result [cfg80211]) from [<bf0dda00>] (cfg80211_process_wdev_events+0x138/0x1c0 [cfg80211])
[ 3726.452052]  r7:d8dd6024 r6:d8dd6004 r5:80000013 r4:c5343a00
[ 3726.460498] [<bf0dd8c8>] (cfg80211_process_wdev_events [cfg80211]) from [<bf0ddac8>] (cfg80211_process_rdev_events+0x40/0x98 [cfg80211])
[ 3726.473127]  r10:d88bc0d8 r9:00000000 r8:00000000 r7:d948ae00 r6:00000040 r5:d88bc420
[ 3726.481129]  r4:d8dd6004
[ 3726.486396] [<bf0dda88>] (cfg80211_process_rdev_events [cfg80211]) from [<bf0d71c4>] (cfg80211_event_work+0x24/0x2c [cfg80211])
[ 3726.498184]  r5:c772d600 r4:d88bc0d4
[ 3726.503264] [<bf0d71a0>] (cfg80211_event_work [cfg80211]) from [<c003ddd4>] (process_one_work+0x1c8/0x470)
[ 3726.513197]  r5:c772d600 r4:d88bc0d4
[ 3726.516863] [<c003dc0c>] (process_one_work) from [<c003e0c4>] (worker_thread+0x48/0x52c)
[ 3726.525151]  r10:d940d200 r9:00000088 r8:c0a3c760 r7:d940d214 r6:c772d614 r5:d940d200
[ 3726.533151]  r4:c772d600
[ 3726.535756] [<c003e07c>] (worker_thread) from [<c00434cc>] (kthread+0x120/0x15c)
[ 3726.543328]  r10:d0067e88 r9:d8ef3f98 r8:c772d600 r7:d94ea000 r6:00000000 r5:c502c460
[ 3726.551332]  r4:d8ef3f80
[ 3726.553933] [<c00433ac>] (kthread) from [<c00090ac>] (ret_from_fork+0x14/0x28)
[ 3726.561319] Exception stack(0xd94ebfb0 to 0xd94ebff8)
[ 3726.566462] bfa0:                                     00000000 00000000 00000000 00000000
[ 3726.574824] bfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 3726.583181] bfe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 3726.589916]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c00433ac
[ 3726.597913]  r4:c502c460
[ 3726.600531] ---[ end trace 53428b45b18f1d67 ]---
[ 4075.415726] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 4075.423088] ieee80211 phy0: brcmf_cfg80211_scan: Connecting: status (3)
[ 4075.707740] ------------[ cut here ]------------
[ 4075.713868] WARNING: CPU: 0 PID: 297 at net/wireless/sme.c:756 __cfg80211_connect_result+0x41c/0x4d0 [cfg80211]
[ 4075.724269] Modules linked in: ipv6 nf_defrag_ipv6 brcmfmac brcmutil sha256_generic libsha256 cfg80211 rfkill snd_soc_simple_card snd_soc_simple_card_utils snd_soc_max98357a snd_soc_bcm2835_i2s regmap_mmio snd_soc_core snd_compress snd_pcm_dmaengine snd_pcm snd_timer snd
[ 4075.748795] CPU: 0 PID: 297 Comm: kworker/u2:1 Tainted: G        W         5.4.51 #1
[ 4075.756666] Hardware name: BCM2835
[ 4075.761541] Workqueue: cfg80211 cfg80211_event_work [cfg80211]
[ 4075.767595] Backtrace:
[ 4075.770129] [<c00156e8>] (dump_backtrace) from [<c0015a34>] (show_stack+0x20/0x24)
[ 4075.777886]  r7:000002f4 r6:bf10d624 r5:00000009 r4:bf135900
[ 4075.783669] [<c0015a14>] (show_stack) from [<c0736d54>] (dump_stack+0x20/0x28)
[ 4075.791085] [<c0736d34>] (dump_stack) from [<c00239a4>] (__warn+0xd0/0x104)
[ 4075.798226] [<c00238d4>] (__warn) from [<c0023d58>] (warn_slowpath_fmt+0x6c/0xc4)
[ 4075.805843]  r7:bf10d624 r6:000002f4 r5:bf135900 r4:00000000
[ 4075.813019] [<c0023cf0>] (warn_slowpath_fmt) from [<bf10d624>] (__cfg80211_connect_result+0x41c/0x4d0 [cfg80211])
[ 4075.823577]  r8:d8dd6084 r7:d8ea1e64 r6:00000000 r5:d8dd6004 r4:d8ea660c
[ 4075.833125] [<bf10d208>] (__cfg80211_connect_result [cfg80211]) from [<bf0dda00>] (cfg80211_process_wdev_events+0x138/0x1c0 [cfg80211])
[ 4075.845621]  r7:d8dd6024 r6:d8dd6004 r5:80000013 r4:d8ea6600
[ 4075.854111] [<bf0dd8c8>] (cfg80211_process_wdev_events [cfg80211]) from [<bf0ddac8>] (cfg80211_process_rdev_events+0x40/0x98 [cfg80211])
[ 4075.866698]  r10:d88bc0d8 r9:00000000 r8:00000000 r7:d948ae00 r6:00000040 r5:d88bc420
[ 4075.874702]  r4:d8dd6004
[ 4075.879969] [<bf0dda88>] (cfg80211_process_rdev_events [cfg80211]) from [<bf0d71c4>] (cfg80211_event_work+0x24/0x2c [cfg80211])
[ 4075.891798]  r5:c772d5a0 r4:d88bc0d4
[ 4075.896834] [<bf0d71a0>] (cfg80211_event_work [cfg80211]) from [<c003ddd4>] (process_one_work+0x1c8/0x470)
[ 4075.906765]  r5:c772d5a0 r4:d88bc0d4
[ 4075.910472] [<c003dc0c>] (process_one_work) from [<c003e0c4>] (worker_thread+0x48/0x52c)
[ 4075.918757]  r10:d940d200 r9:00000088 r8:c0a3c760 r7:d940d214 r6:c772d5b4 r5:d940d200
[ 4075.926717]  r4:c772d5a0
[ 4075.929359] [<c003e07c>] (worker_thread) from [<c00434cc>] (kthread+0x120/0x15c)
[ 4075.936891]  r10:d0067e88 r9:d8fa4b98 r8:c772d5a0 r7:d8ea0000 r6:00000000 r5:c502c6c0
[ 4075.944892]  r4:d8fa4b80
[ 4075.947525] [<c00433ac>] (kthread) from [<c00090ac>] (ret_from_fork+0x14/0x28)
[ 4075.954872] Exception stack(0xd8ea1fb0 to 0xd8ea1ff8)
[ 4075.960063] 1fa0:                                     00000000 00000000 00000000 00000000
[ 4075.968425] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 4075.976743] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 4075.983516]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c00433ac
[ 4075.991514]  r4:c502c6c0
[ 4075.994097] ---[ end trace 53428b45b18f1d68 ]---
@jaroslawprzybylowicz
Copy link

@jaroslawprzybylowicz jaroslawprzybylowicz commented Aug 2, 2020

I'm seeing the same issue with my Raspberry PI Zero W.

Linux luca1 5.4.51+ #1327 Thu Jul 23 10:53:06 BST 2020 armv6l GNU/Linux
brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM43430/1 wl0: Oct 22 2019 01:59:28 version 7.45.98.94 (r723000 CY) FWID 01-3b33decd
@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Aug 8, 2020

I decided to do more debugging myself using modprobe brcmfmac debug=0x14dd36 and it seems I was able to capture the moment wifi stopped working. https://gist.github.com/riptidewave93/787ccd6ef50a7bb0f804d330d0dff33c

Note this was on Linux embedded 5.7.9 #1 Sat Aug 8 13:21:12 CDT 2020 armv6l GNU/Linux which is based off of the rpi 5.7 branch as of commit 95e1914

From the gist, you can see that wifi started failing around 330.527497 when brcmf_sdio_bus_watchdog is first referenced. After that, you see that txdata slows down to almost nothing and multiple calls over and over to brcmf_sdio_bus_watchdog. Digging in the code, this function is called by https://github.com/raspberrypi/linux/blob/rpi-5.7.y/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c#L4045-L4069 It's also worth noting this watchdog code, according to git blame, was last changed 6~ years ago.

This makes me think there may be an issue with the SDIO bus, but I am personally not skilled enough to dig much deeper than this. Could this maybe be a clock issue?

@pelwell Would love your thoughts on this one 😅

@jeremyn54
Copy link

@jeremyn54 jeremyn54 commented Aug 9, 2020

I thought it worth mentioning, although this is not a long-term solution, but for anyone who is looking for a workaround:

If you have already upgraded your WiFi firmware, try:
pi@raspberrypi:~ $ wget http://archive.raspberrypi.org/debian/pool/main/f/firmware-nonfree/firmware-brcm80211_20190114-1+rpt4_all.deb
pi@raspberrypi:~ $ sudo dpkg -i firmware-brcm80211_20190114-1+rpt4_all.deb
pi@raspberrypi:~ $ sudo reboot

If you haven't upgraded your firmware, but would like to continue with the latest OS updates:
pi@raspberrypi:~ $ sudo apt update
pi@raspberrypi:~ $ sudo apt list --upgradeable | grep firmware-brcm80211

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

firmware-brcm80211/testing 1:20190114-1+rpt7 all [upgradable from: 1:20190114-1+rpt4]

Note you will see above the firmware version that would otherwise be installed, then:
pi@raspberrypi:~ $ sudo apt-mark hold firmware-brcm80211

And check that it was successful:
pi@raspberrypi:~ $ apt-mark showhold
firmware-brcm80211

Now it is safe to perform a full upgrade leaving the WiFi function intact:
pi@raspberrypi:~ $ sudo apt -y upgrade

If at anytime it is necessary to un-set the hold on the package to do more testing, etc:
pi@raspberrypi:~ $ sudo apt-mark unhold firmware-brcm80211

I can confirm through quite extensive testing that 20190114-1+rpt4 package version seems very stable with hostapd and other functions. For now it seems to be working fine with the latest kernel.

@robgil
Copy link

@robgil robgil commented Aug 10, 2020

Per @jeremyn54, this seems to have helped me. I've been running this for a few days now and so far no drops. I ended upgrading the firmware as well as the kernel.

root@raspberrypi:~# dpkg -l |grep firmware-brcm80211
ii  firmware-brcm80211                    1:20190114-1+rpt7                      all          Binary firmware for Broadcom/Cypress 802.11 wireless cards
Linux raspberrypi 5.4.51-v7+ #1327 SMP Thu Jul 23 10:58:46 BST 2020 armv7l GNU/Linux
ii  raspberrypi-kernel                    1.20200723-1                           armhf        Raspberry Pi bootloader

Hopefully this helps others. I'll post back if I get any lockups/drops. I'm running it in AP mode.

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Aug 11, 2020

Based on what was shared by @jeremyn54 and @robgil, I extracted the firmware blobs from both of the mentioned raspbian packages:

7.45.98.38 - 20190114-1+rpt4
7.45.98.94 - 20190114-1+rpt7

And on my kernel, Linux buildroot 5.7.9 #1 Mon Aug 10 19:06:58 CDT 2020 armv6l GNU/Linux, I am still seeing the WiFi crash when SCPing large files to the Pi Zero as mentioned earlier.

@lategoodbye
Copy link
Contributor

@lategoodbye lategoodbye commented Aug 22, 2020

There is a promising feature "reset SDIO bus on a firmware crash" in the upcoming Linux 5.9.

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Aug 22, 2020

There is a promising feature "reset SDIO bus on a firmware crash" in the upcoming Linux 5.9.

Sadly I cherry picked this and tested it, as well as a handful of other upcoming patches for 5.9 with no success. The issue doesn't seem to be a firmware crash, but something actually wrong with the SDIO bus from my testing. Really wish this issue would get more eyes from RaspberryPi.

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Aug 27, 2020

As an update to the issue, it seems the cause of the crash, at least in my case, is due to my Pi Zero being connected to a network that has 802.11r fast roaming enabled. If I reconnect to a non 802.11r network, I do not have connectivity issues. I have tested with roamoff=1 as well as roamoff=0, and I can always re-create the driver issue during an inbound SCP to the device. Since roamoff has no impact on the issue, this leads me to think the issue is within the brcmfmac driver on handling 802.11r networks.

@jaroslawprzybylowicz
Copy link

@jaroslawprzybylowicz jaroslawprzybylowicz commented Aug 28, 2020

I can confirm that disabling fast roaming in my AP worked around the problem. I haven't seen connectivity drop ever since.

@riptidewave93
Copy link
Contributor

@riptidewave93 riptidewave93 commented Sep 8, 2020

@jaroslawprzybylowicz I am trying to get more information on what may be causing the issue. Care if I ask what type of AP you are using, and what type of radios it has?

I am personally using a few Ubiquiti Unifi NanoHDs, which use the MediaTek MT7603 for the B/G/N radio.

@DEvil0000
Copy link

@DEvil0000 DEvil0000 commented Sep 8, 2020

@jaroslawprzybylowicz
Copy link

@jaroslawprzybylowicz jaroslawprzybylowicz commented Sep 8, 2020

@riptidewave93 My setup is single UniFi AP-AC-Pro with Qualcomm Atheros QCA9563 onboard. It has both 2.4 and 5GHz radios enabled under the same SSID.

@rraszews
Copy link

@rraszews rraszews commented Sep 8, 2020

For what it's worth, I'm using a TP-Link AC-1750 which has 2.4ghz and 5ghz on different ssids. And I also have only observed the issue when connecting from an android device

@pentlander
Copy link

@pentlander pentlander commented Sep 14, 2020

So on my pi 3B the wifi doesn't die after a while, it doesn't even start up anymore. Here is the output with the debug flag enabled: https://gist.github.com/pentlander/d37fa273f955ac988f71342c47109d28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.