-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted #1403
Comments
|
Since you seem to be able to get failures fairly regularly, can you try enabling some kernel debug output?
Then just carry on as normal. When the brcmfmac firmware dies, capture the output of Since the failure triggers other kernel messages, there is a danger that the useful output is lost before you have chance to capture it. A way to avoid this is to leave a shell constantly saving the kernel messages to a file:
|
here is a syslog (with enabled debug information) of that issue from boot to shutdown/reboot... at 11:34:30 the firmware is halted PS.: i can not force the issue by slowly covering the sensor with a metal case. |
Thanks - I've forwarded your log and background information to Cypress. |
Cypress have responded very quickly with a new firmware to try - there's no indication yet what might have changed.
If you do get another crash, please attach the kernel log as before. |
wow, that is really quick! but maybe it will take quite some time because i haven't figure out how to force the issue manually... so the only thing i can do is waiting... |
Of course - when you are waiting for a failure, no news is good news. |
intermediate report: BTW: the firmware reports: |
Thanks for the update - I'll pass on the message. Cypress didn't state the history of that firmware or what might be different, but it is more recent than anything we have been given to release - you didn't miss it. |
I'm currently experiencing this issue when I connect and eighth device to the WiFi AP. I'm on a Raspberry Pi 4 using hostapd to manage the AP. This solution did not solve the issue:
I attach the kernel log, but in a glance:
Is there any alternative? |
It's a shame you didn't enable debug output from the firmware as I asked the issue creator to do - there is no useful information in the log. I think you've probably reached the limits of the device+firmware combination, but I've asked Cypress how many clients should be able to connect. |
That would be very interesting information. I will run again the issue with debug log activated. |
Although certainly the firmware should not crash under these circumstances, looks like a buffer overflow or bad error handling somewhere. |
I attach a log with debug log activated. The procedure is the same as in my previous comment: |
I see that's with the original firmware. Could you try again with the test firmware? |
Sorry, I changed it back to test a different approach. This is a test applying the change you suggest: |
Thanks - I've passed that (and the other log) on. |
That second log that posted is using the original firmware again - it looks like you've failed to install the updated version. After a successful installation you should get the following result:
If the md5sum doesn't match then you have the wrong file. After booting the kernel log will include:
Go through the installation instructions again, making sure that the download file is in the current directory before you run the
|
FYI, I have been experiencing the same kind of issues, quite reproductible after a few hours of operation. I have updated to the test firmware and enabled the verbose driver logs, will report back if the crash happens again. |
Thanks - testing and feedback like that are invaluable. |
I'm deeply sorry. Somehow I got confused with this similar answer, with a completely different version. The patch you suggest works better. The eighth device is still unable to connect but at least the driver doesn't crash. Now it reports this when the eighth device tries to connect and then it is rejected:
Is it caused by the limit of devices you suggested previously? Thanks in advance |
I think it has to be because of resource limits, but I've not had confirmation of what the limit is supposed to be. I've seen a datasheet suggesting a maximum of 4 AP clients, so 7 isn't bad considering. Out of curiosity, how many devices does your phone's hotspot support? |
In the past I've reached three devices but never explored the limit. I'll try to perform a test later if possible. |
Cypress have confirmed the 7-client limit. |
Ok, that's a pity, but good news anyway. Thank you! |
There's another test firmware - the date is the same as the recent public release but the options list looks different (I have no other information): https://drive.google.com/file/d/10ivocg5PrOwVxAYFKOzJEdv_gdCd-IUF/view?usp=sharing Be sure to rename to brcmfmac43455-sdio.bin when installing it. |
I didn't reproduced my hang issues with the updated firmware. Maybe I didn't wait enough. Anyway, I have updated to the latest firmware and will continue testing:
|
i am wondering, why the new update goes back in time...? the prevous new one: the original one: |
Think you might want to add https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-4.19.y&id=e12bc9cb438d67ddd64c596a907391d008248f7a if you don't want to make the jump to 4.19.129 or .130 from .118. |
I monitor all netdev and linux-wireless posts for anything relevant to both wired and wireless ethernet on the Pi, and pass any that might be of interest along. In the recent batch, we do not think there is anything vital , and I saw nothing about the number of available connections. |
Just fyi, I stumbled upon this ticket while experiencing problems with Wifi disconnecting after longer periods of time (1 - 5 days) on a headless Raspberry PI 4B (up to date Raspbian installed), so I thought I'd give it a try as this is a quite recent conversation. I tried out the Mar 23 firmware (01-88ee44ea) and my Wifi would only connect to the AP on the 5GHz band with unusable low signal strength (and also does not fall back to 2.4GHz). So the Mar 23 one does not seem to be advisable. |
FYI June 25th https://community.cypress.com/docs/DOC-20044
note .214 vs .206 or .202, not saying this will correct any known issue, but it works for me with wlan0 used for upstream and ap0 for hostapd. |
I added debugging with |
@jvonau Is brcmfmac43430 for Raspberry pi3x and brcmfmac43455 for Raspberry pi 4? For raspberry pi 3b+ the instructions(issue - raspberrypi/linux#2453) were for file named brcmfmac43430. Thank you. |
There are a variety of devices on that thread, but 43430 is used on 3B and Zero W, while 43455 is on 3B+, 3A+ and 4B. |
Select the firmware from cypress-firmware-v5.4.18-2020_0625.tar.gz based on what dmesg spits out for you, mine is a rpi4
so I used cyfmac43455-sdio.* files. Remember to back up your old files, rename the new files, reboot and happy testing. |
Thanks @jvonau and @pelwell. dmesg output: So 43455. However, I followed the instructions from raspberrypi/linux#2453 and replaced brcmfmac43430 and surprisingly that resolved the wifi issue I was having on my Raspiberry Pi 3B+. Any clue how that would be possible? |
I tried patching 43455 on my Rpi3B+ and it broke the wifi. I reverted the backup and the wifi is working now. So seems like for Rpi3B+ the file that needs to be replaced is brcmfmac43430(as indicated in raspberrypi/linux#2453). |
As I've said before, there is no 43430 on a 3B+. You can patch the firmware if it makes you feel better, but it won't change anything. Any apparent benefits must be attributed to random chance. |
thanks @pelwell - as you said encountered the issue again today. This time patched with the test firmware which you provided here for 43455 #1403 (comment). Currently the rpi3B+ is working. |
That version of the firmware is now in the RPi-Distro/firmware-nonfree repo and will be in future package and image releases. It's version string is:
|
anyway to know starting which image version/name the change will be bundled? |
recently (2020-07-01) through an upgrade ( 9d5d7f1953769b5ded866217c9afc437
|
It appears that the '.206' version didn't make it into the July 1st release of firmware-brcm80211-20190114-1+rpt7, as the commit with what I assume to be 206 was reverted prior to publishing '+rpt7'. Any idea when this might be released? |
FYI two tables were published earlier today, documenting the very serious downward trend in Raspberry Pi WiFi capacity, as well as some serious crashiness in Raspbian / Raspberry Pi OS's current firmware (7.45.202) for 3 B+ and RPi 4: As it affects those using Raspberry Pi's as learning hotspots / medical hotpots / community hotspots etc. Thanks to the many who contributed to this initial & ongoing analysis, hoping we can all get ourselves out of this ugly rut together over the coming months! |
@holta Worth noting that Cypress may be reallocating ram/resources on the chip not to necessarily enable fancy features but to fix security and/or stability issues in the wifi stack. Thanks for putting together a table of which firmware supports what. For those of us interested primarily in wifi client performance/stability instead of use as an access point, perhaps there could be an option for which firmware is used. (Also, I imagine that since Broadcom sold its wifi IP and product lines to Cypress, that a future RPI device might have other wifi options if they hit an appropriate price/performance point, especially if using Raspberry PIs as wifi access points continues to have popularity.) |
Firstly we haven't asked for any new features from Cypress, we only ask for bug fixes. Whether they have added fancy new features I don't know. We have no access whatsoever to the Cypress source code so are entirely beholden to them to fix issues. This can be a time consuming process. We are a relatively small customer, with little leverage. If anyone has any suggestions how we can put any pressure on them to improve matters, I'd be all ears. Short of moving to another chip supplier with associated HUGE development costs and long (>year) timescales, I cannot see what the way forward is. We do continue to report issues to Cypress, and take what they give us back. I cannot reiterate enough that this change was nothing to do with us - we didn't ask for it and it's as big a surprise to us as it is to you. |
@JamesH65 in order for us to be clear on the implications of using old versions of the firmware is there a list anywhere of CVEs addressed in a given firmware release? |
Not that we maintain. Looking through https://github.com/RPi-Distro/firmware-nonfree/commits/master will get you some of that. |
only as reference related to the limit of number of devices in the WiFi firmware iiab/iiab#2853 |
Describe the bug
hello, i have:
on that RPi i run an access point (hostapd + dnsmasq) on its onboard wifi interface wlan0.
to that access point (wlan0) i connected wifi sensor devices, those are sending its data to my RPi.
so far, so good...
it was working more than 6 month, 24h per day , 7 days a week - only few reboots after bigger updates,
but recently in some situations the wifi interface wlan0 stops working
and only a reboot of the RPi will bring back the wifi adapter back to life.
i think the main cause is the folowing message:
brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted
before it breaks, one of the most distant wifi sensor device with the worst signal was reporting to see the access point with only -85dB...
and on the RPi4 in the /var/log/syslog there is the folowing to see...
this is already the third extra long weekend (because of holiday), that i can observe such an behavoir...
it always starts with a weak signal strength on one of the wifi client.
the client reports -85dB rx signal strength for the wifi access point.
then there is a bunch of messages related to the client with the weak signal:
hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: disassociated
hostapd: wlan0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated
but the communication completely breaks as soon the folowing message appears in the log
brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted
from this point, the log file will be filled with endless messages of:
after that the wifi adapter is not recoverable anymore until i reboot the RPi4.
the wlan0 is dead - no wifi client is able to connect.
a regular normal re-boot after this situation takes for ever, because stopping jobs of wlan0 / hostapd / dhcpsd run into timeouts at least for 3 minutes...
does anybody know, what is going on here?
that is happens mostely on extra long weekends with holidays.
i guess is because there are more wlan activity in the neighborhood speacially at those times.
but why seems the firmware to break in those situation so badly that only a reboot can bring the wifi adapter back to life?
The text was updated successfully, but these errors were encountered: