Raspberry Pi WiFi hotspot firmware reliability fix, incl new/better choices for 3B+ & 4 (WIP, this is PR #3101 rebased) #3103

holta · 2022-01-08T02:03:29Z

Same as PR #3101 but rebased to avoid clutter.

This brings much improved handling of firmware options on Raspberry Pi 3 B+ and 4 especially, and better explanations.

WiFi Hotspot Reliability testing is ongoing but more is needed on the 43430 family (i.e. RPi 3, RPi Zero W, RPi Zero 2 W).

Refs:

Raspberry Pi 3 used to support 32 WiFi connections but is now limited to 10. RPi 3 B+ and RPi 4 are now limited to ~4 [replacing Raspbian with Ubuntu 19.10.1 increases this to 32; what does Ubuntu 20.04 offer?] #823
PR RFC Firmware downgrading [WiFi hotspot capacity workaround, for Raspberry Pi] #2472
What USB 3.0 stick/drive patterns degrade a Raspberry Pi's 2.4GHz WiFi? #2638
Test Raspberry Pi OS 11 on its release. Does RPi WiFi firmware 7.45.18.0 [or its #2853 replacements!] work well? #2820
Opinions about Pi 4B/3B+ WiFi features #2853 (comment)
Number of wireless clients in AP mode is very limited raspberrypi/linux#3010
PR Detect, record if wifi can be AP and suppress installing supporting files #3070
Should host_country_code in local_vars.yml update wpa_supplicant.conf not just hostapd.conf ? #3097

…o rpi-wifi-firmware

…7.45.98.65) ?

tim-moody · 2022-01-09T18:02:46Z

Rather than copying some old version of the firmware to brcmfmacxxxx-sdio.bin.iiab could we start naming them by their version number and use symlinks to activate them, much as @beta-tester suggested on #2853

holta · 2022-01-09T18:17:10Z

Rather than copying some old version of the firmware to brcmfmacxxxx-sdio.bin.iiab could we start naming them by their version number and use symlinks to activate them

That's what this PR does.

Intensive testing continues, over more than 4 days now.

tim-moody · 2022-01-09T18:59:15Z

are you saying that's what this does?

link_fw() {
    if [[ $(readlink /lib/firmware/brcm/$1) != $1.iiab ]] ; then
	echo
	mv /lib/firmware/brcm/$1 /lib/firmware/brcm/$1.$(date +%F-%T)
	ln -s $1.iiab /lib/firmware/brcm/$1
	echo -e "\e[1mSymlinked /lib/firmware/brcm/$1 -> $1.iiab\e[0m"
	touch /tmp/.fw_modified
    fi

what is the .iiab for? why is there a mv?

I thought it's /usr/lib/firmware.
Why isn't the name just brcmfmac43455-sdio.bin_2020-05-22_7.45.214 etc.?
What will https://d.iiab.io/packages/brcmfmac43455-sdio.clm_blob_2021-11-17_rpi be called when there is a new release?

holta · 2022-01-09T20:53:48Z

are you saying that's what this does?

link_fw() {
    if [[ $(readlink /lib/firmware/brcm/$1) != $1.iiab ]] ; then
	echo
	mv /lib/firmware/brcm/$1 /lib/firmware/brcm/$1.$(date +%F-%T)
	ln -s $1.iiab /lib/firmware/brcm/$1
	echo -e "\e[1mSymlinked /lib/firmware/brcm/$1 -> $1.iiab\e[0m"
	touch /tmp/.fw_modified
    fi

what is the .iiab for?

Each .iiab is a symlink that protects the IIAB implementer's choice of firmware (e.g. among ~4 choices, in the case of Raspberry Pi 4 and 3 B+), that points to the actual firmware files chosen during IIAB installation.

This allows /usr/bin/iiab-check-firmware (and iiab-check-firmware.service during bootup) to "instantly" restore the implementer's choice — without slowpoke Ansible bureaucracy — after an apt update.

(In lieu of the heavy-handed apt-mark hold firmware-brcm80211 on Raspberry Pi OS, and apt-mark hold linux-firmware-raspi2 on Ubuntu on Raspberry Pi.)

why is there a mv?

To capture and retain the results of any apt updates. Allowing advanced operators visibility into all future firmware options.

What is moved after an apt update can be a symlink or a file, either way, for safekeeping.

I thought it's /usr/lib/firmware.

It's /lib/firmware/brcm and /lib/firmware/cypress

Why isn't the name just brcmfmac43455-sdio.bin_2020-05-22_7.45.214 etc.? What will https://d.iiab.io/packages/brcmfmac43455-sdio.clm_blob_2021-11-17_rpi be called when there is a new release?

Raspberry Pi Foundation often does not use the canonical firmware files from Broadcom/Cypress/Infineon. These are often customized before being included within Raspberry Pi OS.

So the actual heritage/lineage of firmware files are being indicated (or at least summarized) in the filenames, including the publisher's own timestamp (datestamp) e.g. from GitHub typically, if that's what's available.

If Raspberry Pi OS (or anyone!) later releases demonstrably better firmware files, that will be great.

At that time, these too will be labelled to indicate (summarize) where they came from.

tim-moody · 2022-01-09T21:15:48Z

I thought it's /usr/lib/firmware.

It's /lib/firmware/brcm and /lib/firmware/cypress

root@box:~# ll /lib
lrwxrwxrwx 1 root root 7 Oct 30 12:09 /lib -> usr/lib

Sounds like most of this is to protect against apt update and assumes it will update brcmfmac43455-sdio.bin.

Before iiab install that was a symlink to ../cypress/cyfmac43455-sdio.bin, so I doubt it will get updated.

If instead it was a symlink to brcmfmac43455-sdio.bin_2020-05-22_7.45.214 wouldn't that remain after an update?

Aren't the actual firmware files now in ../cypress? That's where cyfmac43455-sdio-minimal.bin is. So if an implementer wants your 19, just symlink brcmfmac43455-sdio.bin to ../cypress/cyfmac43455-sdio-minimal.bin. I don't think an update will change that.

holta · 2022-01-09T21:47:06Z

The 4 files (or symlinks) that matter to activate WiFi are:

/lib/firmware/brcm/brcmfmac43455-sdio.bin
/lib/firmware/brcm/brcmfmac43455-sdio.clm-blob
/lib/firmware/brcm/brcmfmac43430-sdio.bin
/lib/firmware/brcm/brcmfmac43430-sdio.clm-blob

A crystal ball is (effectively) needed to be able to determine what future apt updates will do to the above 4 files — on every different OS.

e.g. if any OS and its apt updates choose to rename or move /lib/firmware/cypress to /lib/firmware/infineon (the new name of the company, which changed yet again) that is their choice.

Hence the reason the above 4 crucial files/symlinks are verified and re-established by iiab-check-firmware.service during each boot (if both firmware variables specify anything other than "os", in /etc/iiab/local_vars.yml etc).

tim-moody · 2022-01-09T22:07:22Z

Well I just did an iiab install and see

-rw-r--r--  1 root root  488193 Jan  9 14:23 brcmfmac43455-sdio.bin
-rw-r--r--  1 root root  488193 Jan  9 14:23 brcmfmac43455-sdio.bin.iiab
-rw-r--r--  1 root root  637406 Jan  9 14:23 brcmfmac43455-sdio.bin.orig
-rw-r--r--  1 root root    2074 Dec  6 14:10 brcmfmac43455-sdio.txt
lrwxrwxrwx  1 root root      31 Dec  6 14:10 brcmfmac43455-sdio.bin.2022-01-09-14:23:35 -> ../cypress/cyfmac43455-sdio.bin

There's no need for all of that. Just have brcmfmac43455-sdio.bin point to one of

./cypress/cyfmac43455-sdio.bin
./cypress/cyfmac43455-sdio-minimal.bin
./brcmfmac43455-sdio.xxx.bin
./brcmfmac43455-sdio.yyy.bin

where xxx and yyy are the old versions preserved by iiab

A crystal ball is (effectively) needed to be able to determine what future apt updates will do to the above 4 files — on every different OS.

e.g. if any OS and its apt updates choose to rename or move /lib/firmware/cypress to /lib/firmware/infineon (the new name of the company, which changed yet again) that is their choice.

True, but then raspios will break on apt update.

holta · 2022-01-09T23:31:38Z

Just have brcmfmac43455-sdio.bin point to one of

./cypress/cyfmac43455-sdio.bin ./cypress/cyfmac43455-sdio-minimal.bin ./brcmfmac43455-sdio.xxx.bin ./brcmfmac43455-sdio.yyy.bin
where xxx and yyy are the old versions preserved by iiab

I avoided the above approach to make it more resilient after apt updates.

So both (1) implementer's install-time choice of firmware, (encoded in the 4 .iiab symlinks) and (2) a rigorous set of firmware choices — remain explicitly visible to operators within /lib/firmware/* — even after apt updates.

(There have been times in recent years when RasPiOS has accidentally pushed a highly imperfect WiFi firmware, so it's best to be prepared for all sorts of similar problems, to be able to revert flexibly and quickly.)

e.g. if any OS and its apt updates choose to rename or move /lib/firmware/cypress to /lib/firmware/infineon (the new name of the company, which changed yet again) that is their choice.

True, but then raspios will break on apt update.

My point was that any OS (including RasPiOS itself) can change most any file/symlink in /lib/firmware/* at any time, upon any subsequent apt update.

Wiping away cypress/cyfmac43455-sdio.bin or whatever.

So best to be prepared, allowing for nearly instant and easy reverting.

ASIDE/FWIW: Linux's "5.15" kernel's support for multiple / versioned WiFi firmwares...might be implemented by RasPiOS and/or other OS's to change this layout in coming years...to deal with situations exactly like this.

tim-moody · 2022-01-10T14:05:54Z

My point was that any OS (including RasPiOS itself) can change most any file/symlink in /lib/firmware/* at any time, upon any subsequent apt update.
Wiping away cypress/cyfmac43455-sdio.bin or whatever.

and mine was that that would break raspios as they currently use symlinks from brcm to cypress as a key part of their strategy. to change that link would be highly reckless and would only be undertaken when there is a new release of raspios at which point all of this would be broken.

that's definitely a glass half empty approach. it says the snapshot you took is guaranteed to be better than current shipping version.

tim-moody · 2022-01-10T14:27:32Z

Just to be clear, on installation of raspios brcm/brcmfmac43455-sdio.bin is a symlink to cypress/cyfmac43455-sdio.bin.
What happens if the user choses wifi_hotspot_capacity_rpi_fix: False?
What if the user choses wifi_hotspot_capacity_rpi_fix: True and rpi3bplus_rpi4_wifi_firmware: os?

holta · 2022-01-10T14:41:34Z

Just to be clear, on installation of raspios brcm/brcmfmac43455-sdio.bin is a symlink to cypress/cyfmac43455-sdio.bin. What happens if the user choses wifi_hotspot_capacity_rpi_fix: False? What if the user choses wifi_hotspot_capacity_rpi_fix: True and rpi3bplus_rpi4_wifi_firmware: os?

These questions don't make sense as this PR very intentionally removes variable wifi_hotspot_capacity_rpi_fix.

What matters, as mentioned near the top of this ticket, is that these 4 files (or symlinks) activate WiFi:

/lib/firmware/brcm/brcmfmac43455-sdio.bin
/lib/firmware/brcm/brcmfmac43455-sdio.clm-blob
/lib/firmware/brcm/brcmfmac43430-sdio.bin
/lib/firmware/brcm/brcmfmac43430-sdio.clm-blob

Every apt WiFi firmware package from RaspiOS or Ubuntu to date makes adjustments to those 4 files-or-symlinks, as these are the 4 files/symlinks that matter. Just the same as IIAB has done for about 1.5 years now, and will continue to do.

I apologize I do not understand any of the above claims about breaking RasPiOS or any other OS. (But insofar as I can tell, they appear to be based on extreme hypotheticals.)

…arning

tim-moody · 2022-01-10T14:50:43Z

this PR very intentionally removes variable wifi_hotspot_capacity_rpi_fix

sorry missed that.

so does rpi3bplus_rpi4_wifi_firmware: os then do nothing?

holta · 2022-01-10T15:30:31Z

so does rpi3bplus_rpi4_wifi_firmware: os then do nothing?

Essentially that's correct. The details are:

During IIAB installation, it takes a backup snapshot copy of the existing 4 WiFi firmware files (2 for the 43430 + 2 for the 43455) to /lib/firmware/brcm/*.orig to protect the operator from apt updates.
During IIAB installation, *.iiab become symlinks to each pair of *.orig files (if either chip architecture's var is set to "os") to encode the implementer's intention, allowing for quick recovery if later necessary.
During every boot (and when it's run manually) /usr/bin/iiab-check-firmware does NOT set the pair of high-level symlinks to *.iiab in this one case (i.e. if rpi3bplus_rpi4_wifi_firmware: os, or likewise if rpizerow_rpi3_wifi_firmware: os). WiFi firmware is intentionally allowed to drift forward with the OS's apt updates, in this one case, i.e. if the IIAB implementer has chosen the "os" setting for one and/or the other chip family.
An advanced operator who is suddenly later unhappy with the OS's newly apt updated firmware, then has the option to manually set the pair of high-level symlinks in /lib/firmware/brcm to /lib/firmware/brcm/*.iiab for their chip — if RasPiOS-or-any-other-OS's accidentally push bad firmware — as has repeatedly happened in the past.

tim-moody · 2022-01-10T16:12:00Z

During IIAB installation, it takes a backup snapshot copy of the existing 4 WiFi firmware files (2 for the 43430 + 2 for the 43455) to /lib/firmware/brcm/*.orig to protect the operator from apt updates.

the ones in cypress? nothing is removing or changing the cypress file, so what is the need to preserve it?

During every boot

where is this done? are you saying that on some occasions a script named check makes a change?

pair of high-level symlinks'

are from what to what?

What setting do I use if I don't want any of the 4 steps? What setting do I use if I just want the links from brcm to cypress minimal?

holta · 2022-01-10T17:42:31Z

During IIAB installation, it takes a backup snapshot copy of the existing 4 WiFi firmware files (2 for the 43430 + 2 for the 43455) to /lib/firmware/brcm/*.orig to protect the operator from apt updates.

the ones in cypress?

Yes.

nothing is removing or changing the cypress file, so what is the need to preserve it?

Your assumption is incorrect. Every OS overwrites /lib/firmware/cypress/cypress/cyfmac43455-sdio.bin (etc, and the links to them) during apt updates at the moment, and again this is precisely why these 2+2 are being copied to *.orig during IIAB installation.

As Raspberry Pi's Phil Elwell has mentioned, the 5.15 LTS kernel released 2021-10-31 begins to address this core issue, allowing more flexible firmware filenames, so we might see that in coming years, if diverse OS's choose to implement this (hopefully not each in their own way!)

During every boot

where is this done? are you saying that on some occasions a script named check makes a change?

As has been the case for about 1.5 years now, yes.

Recall that during on every boot iiab-check-firmware.service invokes /usr/bin/iiab-check-firmware to do exactly this.

And again: if firmware vars are set to "os" it intentionally allows ongoing/future WiFi firmware apt updates to take effect (for that chip family).

pair of high-level symlinks'

are from what to what?

The 2 pairs of high-level /lib/firmware/brcm files/symlinks (that's the 4 critical files/symlinks, that have been repeatedly listed above) are linked to *.iiab when the firmware vars as not set to "os".

RECAP EXAMPLES:

Set rpi3bplus_rpi4_wifi_firmware: os and these will not be modified:
- /lib/firmware/brcm/brcmfmac43455-sdio.bin
- /lib/firmware/brcm/brcmfmac43455-sdio.clm-blob
At the same time these "orphan" symlinks -> backup copy snapshots are set up to allow for quick (manual) recovery from future apt updates:
- /lib/firmware/brcm/brcmfmac43455-sdio.bin.iiab -> /lib/firmware/brcm/brcmfmac43455-sdio.bin.orig
- /lib/firmware/brcm/brcmfmac43455-sdio.clm-blob.iiab -> /lib/firmware/brcm/brcmfmac43455-sdio.clm-blob.orig
Set rpizerow_rpi3_wifi_firmware: 30 and these "non-orphan" symlinks are set up to allow for quick (automatic) recovery from future apt updates:
- /lib/firmware/brcm/brcmfmac43430-sdio.bin -> /lib/firmware/brcm/brcmfmac43430-sdio.bin.iiab -> /lib/firmware/brcm/brcmfmac43430-sdio.bin_2018-09-11_7.45.98.65
- /lib/firmware/brcm/brcmfmac43430-sdio.clm-blob -> /lib/firmware/brcm/brcmfmac43430-sdio.clm-blob.iiab -> /lib/firmware/brcm/brcmfmac43430-sdio.clm_blob_2018-09-11_7.45.98.65

tim-moody · 2022-01-10T18:47:39Z

OK. I'll wait for the movie.

holta · 2022-01-10T19:29:49Z

FYI

I tested this PR repeatedly on both RPi 3 and Zero W — each with several tests of both of these firmwares — all of which thankfully appear to work fine:
- rpizerow_rpi3_wifi_firmware: os (with current RasPiOS) activates 7.45.98.118 — published by RPi on 2021-11-17.
- rpizerow_rpi3_wifi_firmware: 30 (on any OS) activates 7.45.98.65 — published by Broadcom/Cypress on 2018-09-11.
I'm going to leave the default as rpizerow_rpi3_wifi_firmware: os for now, unless there are strong objections?

As this appears to offer about 10 WiFi client device connections, as well as ~3 more years of security updates since 2018.

So any implementer that's intending their Raspberry Pi Zero W, Zero 2 W or 3 to be used in busy classrooms (or busy hospitals, etc!) should use rpizerow_rpi3_wifi_firmware: 30 in their /etc/iiab/local_vars.yml to allow for 30 WiFi client devices. And if the IIAB is already deployed, of course run cd /opt/iiab/iiab && sudo ./runrole firmware && sudo reboot to enact that!
Anybody that installs IIAB with the firmware var (for their Raspberry Pi hardware) set to "os" is choosing to take the risk of all future apt updates — which typically means a few more security updates every year AND a couple fewer WiFi student connections being possible every year.

(Not that apt updates are common in the field. But any field operator that actually puts their IIAB online, and then apt updates it every year or so, does so entirely at their own risk — with the relevant apt packages in this case being firmware-brcm80211 on Raspberry Pi OS and apt package linux-firmware-raspi2 on Ubuntu.)

holta · 2022-01-10T19:47:16Z

It's time to expand community testing of this PR by merging it into master.

Thank you to everyone who is able to help!

Aside: This can and will be backported into IIAB 7.2 later, if all looks good.

holta · 2022-12-29T14:15:25Z

FYI, arising from 2022-06-13:

In an up-to-date Raspberry Pi OS install, you can switch between the two
variants by running the following command:

    sudo update-alternatives --config cyfmac43455-sdio.bin

This method will persist across firmware-brcm80211 updates.

Per:

RPi-Distro/firmware-nonfree@ef28cdc

Thanks to @martignoni:

Update wireless chip firmware swapping logic moodlebox/moodlebox#280

Tangentially related:

Make wifi unblocking instructions more generic moodlebox/moodlebox#279
Should IIAB run 'rfkill unblock wlan' ? This worked to get IIAB running on Mint 21 on Acer TravelMate P643-M #3398

holta and others added 5 commits January 7, 2022 01:59

WIP: RPi WiFi firmware cleanup/options, espec for 3B+ & 4

64efea0

Clarify RPi WiFi firmware capacity options

81790ba

Merge branch 'rpi-wifi-firmware' of https://github.com/holta/iiab int…

56dc06b

…o rpi-wifi-firmware

roles/firmware: code + UI cleanup (7.45.98.65 needs testing on RPi 3/W)

fbe585d

TBD: Default RPi 3/W to ~10 connections (7.45.98.118) instead of 30 (…

4ce5568

…7.45.98.65) ?

holta added the bug label Jan 8, 2022

holta added this to the 8.0 milestone Jan 8, 2022

holta added 5 commits January 7, 2022 21:14

firmware/tasks/download.yml: Touchup clarifs

029e169

firmware/tasks/download.yml: Clarify firmware origins

6c31a2a

/usr/bin/iiab-check-firmware: Instructions (WIP!)

4cb5a30

firmware/tasks/main.yml: Context + clarifs

8b07f87

firmware/tasks/main.yml: Mention USB 3.0 collision w/ 2.4 GHz WiFi

9ec670b

holta added 2 commits January 9, 2022 21:07

scripts/iiab-diagnostics: List *43430* and *43455* in /lib/firmware/brcm

14e249e

Update iiab-diagnostics.README.md

a85ce3c

firmware/templates/iiab-firmware-warn.sh: Clarify /tmp/.fw_modified w…

dd2bf38

…arning

holta merged commit a2300c3 into iiab:master Jan 10, 2022

holta added a commit that referenced this pull request Jan 10, 2022

firmware/tasks/main.yml: Explain context @ #2853 & #3103

3059572

holta mentioned this pull request Apr 5, 2022

Test Raspberry Pi OS 11 on its release. Does RPi WiFi firmware 7.45.18.0 [or its #2853 replacements!] work well? #2820

Closed

holta mentioned this pull request Feb 25, 2023

Set WiFi firmware in /lib/firmware/cypress due to RasPiOS & Ubuntu changes #3482

Merged

holta mentioned this pull request Dec 18, 2023

rpi w2 and alternative wifi firmware #3686

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Raspberry Pi WiFi hotspot firmware reliability fix, incl new/better choices for 3B+ & 4 (WIP, this is PR #3101 rebased) #3103

Raspberry Pi WiFi hotspot firmware reliability fix, incl new/better choices for 3B+ & 4 (WIP, this is PR #3101 rebased) #3103

holta commented Jan 8, 2022 •

edited

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022 •

edited

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022 •

edited

tim-moody commented Jan 10, 2022

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 •

edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 •

edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 •

edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Dec 29, 2022 •

edited

Raspberry Pi WiFi hotspot firmware reliability fix, incl new/better choices for 3B+ & 4 (WIP, this is PR #3101 rebased) #3103

Raspberry Pi WiFi hotspot firmware reliability fix, incl new/better choices for 3B+ & 4 (WIP, this is PR #3101 rebased) #3103

Conversation

holta commented Jan 8, 2022 • edited

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022 • edited

tim-moody commented Jan 9, 2022

holta commented Jan 9, 2022 • edited

tim-moody commented Jan 10, 2022

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 • edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 • edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 • edited

tim-moody commented Jan 10, 2022

holta commented Jan 10, 2022 • edited

holta commented Jan 10, 2022 • edited

holta commented Dec 29, 2022 • edited

holta commented Jan 8, 2022 •

edited

holta commented Jan 9, 2022 •

edited

holta commented Jan 9, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Jan 10, 2022 •

edited

holta commented Dec 29, 2022 •

edited