This is heavily based on this page with some tweaks to make it boot exactly the way I want it to and configure the network like I wanted. Also, lots of things are "reverse engineered" from ClusterHAT meaning I have one and saw how it works.
Note: These are the working notes of how I did it. I could have just done it but taking notes is good because:
The difference between screwing around and science is writing it down -- Adam Savage
After all this is done, we should have one or more raspberry pi zeros booting
- Off NFS without SD cards
- Using network via USB-gadget-ethernet
- With network configured to be part of the local LAN via a bridge
Configure a range for static IP addresses. How to do that depends on your
setup, but usually implies logging into your router and changing its DHCP
server configuration. I have the network 192.168.0.0/24 and saved
IPs 201 ... 254 for static addresses.
Again, details on how to do these things will vary depending on your Linux distribution or whatever operating system you are using (I don't expect anything that is not Linux will work)
- Install a NFS server
- Install USBBoot
- Install bridge-utils
- Connect yourself to the network via ethernet (it can probably be made to work with wifi)
- Enable IP forwarding and set your IPTables FORWARD policy to accept.
Details may vary but we want an ethernet bridge that includes your ethernet interface.
For example, mine is enp4s0f3u1u4 and my bridge is br0so:
If you are using NetworkManager, it has a GUI to configure it that works just fine.
This is the bridge to which we will also connect the zeros' ethernets so they become part of the LAN.
$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.a2d599084fcf yes enp4s0f3u1u4If all went well, you will have an IP address on the br0 interface and not in your "real" ethernet:
$ ifconfig -a
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.72 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 2800:810:505:2c8::1004 prefixlen 128 scopeid 0x0<global>
inet6 fe80::ef7a:62ed:f9:5ce5 prefixlen 64 scopeid 0x20<link>
inet6 2800:810:505:2c8:2dff:72aa:2fce:b293 prefixlen 64 scopeid 0x0<global>
ether a2:d5:99:08:4f:cf txqueuelen 1000 (Ethernet)
RX packets 82794 bytes 123703379 (117.9 MiB)
RX errors 0 dropped 178 overruns 0 frame 0
TX packets 77001 bytes 49355088 (47.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp4s0f3u1u4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 28:ee:52:1a:8c:20 txqueuelen 1000 (Ethernet)
RX packets 133110 bytes 125755066 (119.9 MiB)
RX errors 0 dropped 78 overruns 0 frame 0
TX packets 66604 bytes 10713246 (10.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0There are two ways to do this, the easy way and the hard way. You choose.
We will boot a raspberry pi zero, configure it, and then turn its filesystem into a NFS root. Since we will know exactly what we changed we can then use that as the basis for just copying and modifying it for as many machines as we want, which is the easy way. So, you could just skip this section and do what's described in the next one. You would be, of course, a chicken.
Hello non-feathered readers, let's do this.
-
Get raspberry pi OS 32-bit light (or whatever) into a SD card. Doesn't need to be a fast or large SD card, since we will only use it for a few minutes.
If you will try to do this using something other than raspberry pi os then the details of how to do it will change, but the concepts should still work.
Now we are going to make some changes in the configuration of that SD card before booting from it.
- Create empty
boot/sshfile so it has SSH enabled on boot. - Edit
boot/config.txtand add this at the bottom:
# enable OTG
dtoverlay=dwc2
# set initramfs
initramfs initrd.img followkernel
- Edit
boot/cmdline.txtand add this at the end of the 1st line:
modules-load=dwc2,g_ether
- Edit
etc/network/interfaces.d/usb0to configure itsusb0interface:
auto usb0
allow-hotplug usb0
iface usb0 inet static
address 192.168.0.201/24
gateway 192.168.0.1
dns-nameservers 8.8.8.8 1.1.1.1 9.9.9.9
- Enable ssh for the future
sudo systemctl enable ssh
At this point we should be able to boot off this SD card, so insert the SD card into the Zero, plug the Zero into your computer via USB, plug some monitor in its HDMI interface and let's see what happens.
The first boot is pretty slow and convoluted because it will do things like resize the filesystem to use the whole SD card and whatnot. But eventually it should finish.
At some point, your computer (not the Zero!) will have a new network interface.
In my case it's called enp4s0f3u1u1
NOTE: The actual USB port on which you plug it is important, because it determines the name of the ethernet interface to which your Zero is connected, so ... take notes.
We want that ethernet interface to be added to the bridge, so ... make it so
$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.a2d599084fcf yes enp4s0f3u1u4
enp4s0f3u1u1Reboot the Zero, and now it should boot with the static IP we configured and say something like
"My IP address is 192.168.0.201"
If everything went particularly well you should even be able to SSH into it!
$ ssh pi@192.168.0.201
The authenticity of host '192.168.0.201 (192.168.0.201)' can't be established.
ED25519 key fingerprint is SHA256:gfs9NKRE1y7Oy0lrA3F9dcXg56JEmN0yyFRoAo8o86M.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.201' (ED25519) to the list of known hosts.
pi@192.168.0.201's password:
Linux raspberrypi 5.10.17+ #1414 Fri Apr 30 13:16:27 BST 2021 armv6l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
SSH is enabled and the default password for the 'pi' user has not been changed.
This is a security risk - please login as the 'pi' user and type 'passwd' to set a new password.
pi@raspberrypi:~$ Next step is to create a ramdisk that has enough "stuff" (technical term) in it to boot without the SD card.
Edit /etc/initramfs-tools/modules so it has at least this:
g_ether
libcomposite
u_ether
udc-core
usb_f_rndis
usb_f_ecm
Create the ramdisk:
$ sudo update-initramfs -c -k `uname -r`
update-initramfs: Generating /boot/initrd.img-5.10.17+
$ sudo mv /boot/initrd.img-5.10.17+ /boot/initrd.img
Reboot one last time to make sure everything works.
Now we will take this SD card, and turn it into a NFS root so we can boot our Zero off it without any onboard storage.
- Take the SD card and put it back in your conmputer.
As root, create a "p1" folder somewhere and copy the contents of the SD card in it preserving permissions and such:
# mkdir -p /zeros/p1
# cp -a /run/[whatever]/rootfs/* /zeros/p1/
# cp -a /run/[whatever]/boot/* /zeros/p1/boot/
We'll need to configure a few things to make the Zero boot with root on NFS:
- Change
/root/p1/boot/cmdline.txtto look more or less like this (adjust as needed):
console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.0.72:/zeros/p1 rw elevator=deadline fsck.mode=skip rootwait modules-load=dwc2,g_ether ip=192.168.0.201:192.168.0.1::255.255.255.0:p1:usb0:static rootwait
Explanation:
-
Set root to be nfs
-
Set nfsroot to
{the ip of your NFS server}:/zeros/p1 -
Don't try to FSCK a NFS server
-
Configure
usb0network interface to be static, IP is192.168.0.201, etc. -
Share
/zeros/p1via NFS with the IP of the zero by adding this in/etc/exports:
/zeros/p1 192.168.0.201(rw,async,insecure,no_subtree_check,no_root_squash)
Then run exportfs -arv
- Edit
/zeros/p1/etc/fstaband remove references to local devices. That usually means that the only uncommented line will be the one aboutproc. Mine looks like this:
proc /proc proc defaults 0 0
# a swapfile is not a swap partition, no line here
# use dphys-swapfile swap[on|off] for that
At this point, your Zero should be able to boot without a SD card if we give it a little help. Plug it in (without SD card, of course) and run this:
$ sudo rpiboot -v -d /zeros/p1/boot/
It should start telling you how it's sending files to a device. What's happening is that
rpiboot is giving your Zero the files it needs to start booting, such as the kernel, the ramdisk and so on. After a while it should be fully booted, with network and SSH active.
At this point we have a fully functional and confitured NFS root tree. Let's save it.
As root:
# cd /zerros/p1
# tar cfJ ~/pi1.tar.xz .
You will now have a fully-configured image in ~/pi1.tar.xz. I have made mine available here
If you have read "The Hard Way" just smirk condescendingly and move along.
- Download pi1.tar.xz
As root in your machine:
# mkdir -p /zeros/p1
# cd /zeros/p1
# tar xvf /wheverver/you/put/it/pi1.tar.xz
-
Choose a static IP for the Zero (I used 192.168.0.201)
-
Edit the following files to make sure they agree with your network configuration:
-
/zeros/p1/boot/cmdline -
/zeros/p1/etc/network/interfaces.d/usb0 -
Export
/zeros/p1via NFS for the IP of the Zero by putting something like this in/etc/exports
/zeros/p1 192.168.0.201(rw,async,insecure,no_subtree_check,no_root_squash)
Then run exportfs -arv
At this point, your Zero should be able to boot without a SD card if we give it a little help. Plug it in (without SD card, of course) and run this:
$ sudo rpiboot -v -d /zeros/p1/boot/
It should start telling you how it's sending files to a device. What's happening is that
rpiboot is giving your Zero the files it needs to start booting, such as the kernel, the ramdisk and so on. After a while it should be fully booted, but the network won't work.
At some point, your computer (not the Zero!) will have a new network interface.
In my case it's called enp4s0f3u1u1
NOTE: The actual USB port on which you plug it is important, because it determines the name of the ethernet interface to which your Zero is connected, so ... take notes.
We want that ethernet interface to be added to the bridge, so ... make it so
$ brctl show
bridge name bridge id STP enabled interfaces
br0 8000.a2d599084fcf yes enp4s0f3u1u4
enp4s0f3u1u1Reboot the Zero, and now it should boot with the static IP we configured and say something like
"My IP address is 192.168.0.201"
If everything went particularly well you should even be able to SSH into it!
Congratulations, you made it the easy way. WHICH IS GOOD ENOUGH.
If you have more than one Zero (you should!) then you may want to automate the second one further. Actually, after we are done with this part we'll have something easier than "the easy way".
What's different for the second zero?
- IP is
192.168.0.202instead of192.168.0.201 - Its nfsroot is
/zeros/p2instead of/zeros/p1 - The interface we add to the bridge will be different.
Further, if this was configured in a whole different network and system that's not my own, what would change?
- Default route, DNS servers, network mask are different
- IP address of the NFS server is different
- Path to the nfsroots is different
- Ethernet interface name and bridge name will be different
That's a surprisingly small number of things to change to boot N different systems off a server.
Nothing else?
Well, we'll need to use one other feature of rpiboot so it works correctly for more than one device, but it's easy.
So, we need a simple way to "template" our tarball with the nfsroot and make it accept a few parameters and configure a few files.
Note: We know exactly what files need templating because I took notes. TAKE NOTES WHEN YOU DO STUFF.
The changes we need to do to add a second, third or whatever many Zeros we want are pretty mechanical. So, let's automate them. Or not, feel free to do things by hand like a farmer.
- Template language: jinja2
- Needed tools: j2cli
Not to go into details but here is the file with the configuration data config.yaml you can edit and the only files that need templating are these:
/zeros/pX/boot/cmdline.txttemplated here/zeros/pX/etc/network/interfaces.d/usb0templated here
So, if you expand the tarball as /zeros/p3 you can configure it doing something like this:
$ j2 cmdline.txt.j2 config.yaml -o /zeros/p3/boot/cmdline.txt
$ j2 usb0.j2 config.yaml -o /zeros/p3/etc/network/interfaces/usb0
Now configure /etc/exports and then boot the Zero and do the whole "add interfaces to the bridge" thing.
Currently to boot, say, raspbery pi #3 we need to go to /zeros/pi3/boot and run rpiboot. That is ... not practical.
Luckily, rpiboot supports booting multiple devices independently as long as we know their USB path.
Their what? Their USB path. So, USB is shaped like a tree. Here's mine (yes, my computer is complicated):
$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 10000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 2: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 480M
|__ Port 2: Dev 3, If 0, Class=Wireless, Driver=btusb, 12M
|__ Port 2: Dev 3, If 1, Class=Wireless, Driver=btusb, 12M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 4: Dev 8, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
|__ Port 2: Dev 7, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 4: Dev 4, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 110, If 0, Class=Vendor Specific Class, Driver=, 12M
|__ Port 2: Dev 11, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 20, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 21, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 3: Dev 21, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 4: Dev 22, If 2, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 4: Dev 22, If 0, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 4: Dev 22, If 3, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 4: Dev 22, If 1, Class=Audio, Driver=snd-usb-audio, 12M
|__ Port 2: Dev 23, If 1, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 2: Dev 23, If 2, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 2: Dev 23, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 3: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 45, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 92, If 2, Class=Audio, Driver=snd-usb-audio, 480M
|__ Port 1: Dev 92, If 0, Class=Video, Driver=uvcvideo, 480M
|__ Port 1: Dev 92, If 3, Class=Audio, Driver=snd-usb-audio, 480M
|__ Port 1: Dev 92, If 1, Class=Video, Driver=uvcvideo, 480M
|__ Port 1: Dev 92, If 4, Class=Human Interface Device, Driver=usbhid, 480M
|__ Port 4: Dev 49, If 0, Class=Mass Storage, Driver=usb-storage, 480M
|__ Port 4: Dev 52, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 4, If 1, Class=Video, Driver=uvcvideo, 480M
|__ Port 4: Dev 4, If 0, Class=Video, Driver=uvcvideo, 480M
But don't worry, you don't need to understand all that, you just need to run one command and look carefully.
Unplug your Zero, and now boot it running this command:
rpiboot -vv -d /zeros/p1/boot/
Right before it starts sending files to the Zero it will show something like this:
Found device 1 idVendor=0x0a5c idProduct=0x2763
Bus: 1, Device: 113 Path: 1-1.1
Found candidate Compute Module...Loading: /zeros/p1/boot//bootcode.bin
Device located successfully
See that 1-1.1 ? That's the path (yours will be different)
Now create a /zeros/boot and create in it a link called 1-1.1 pointing to /zeros/p1/boot and repeat for every zero you want to boot like this.
# mkdir /zeros/boot
# cp /zeros/p1/boot/bootcode.bin /zeros/boot/
# ln -s /zeros/p1/boot/ /zeros/boot/1-1.1
It should look like this (with yout own USB paths instead of mine):
# ls -l /zeros/boot/
total 52
lrwxrwxrwx 1 root root 15 Jul 24 17:56 1-1.1 -> /zeros/p1/boot/
lrwxrwxrwx 1 root root 15 Jul 24 18:20 1-1.2 -> /zeros/p2/boot/
-rw-r--r-- 1 root root 52456 Jul 24 17:58 bootcode.bin
Now, if you run rpiboot -v -o -d /zeros/boot -l this will follow the link with the name of the USB path of the device and loop and try again and so on, booting as many Zeros as you need as many times as needed.
You can, of course, make this a systemd service or whatever.