Aarch64 SD images missing pi-zero-2.dtb file require to boot.

[deftdawg]

[Apologies in advance if this is not the correct place to report this issue; there is basically no current documentation that I could find on how to boot and run nix on Raspberry Pi Zero 2]

Attempting to boot the Aarch64 SD card images built by Hydra on a Raspberry Pi Zero 2 fails because uboot cannot find gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom/bcm2837-rpi-zero-2.dtb

Based on this line in master:

nixpkgs/pkgs/os-specific/linux/kernel/linux-rpi.nix

Line 78 in 270de46

copyDTB bcm2710-rpi-zero-2.dtb bcm2837-rpi-zero-2.dtb

It appears the file and a few others should be included, but for whatever reason (maybe this linux-rpi is not used or Hydra building off an different branch?) the file is not present in the image.

Here are the files actually present on the flashed SD card:

ls -1 /run/media/${USER}/NIXOS_SD/boot/nixos/gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom
bcm2711-rpi-4-b.dtb
bcm2711-rpi-400.dtb
bcm2837-rpi-3-a-plus.dtb
bcm2837-rpi-3-b-plus.dtb
bcm2837-rpi-3-b.dtb
bcm2837-rpi-cm3-io3.dtb
bcm4908
northstar2
stingray

Steps To Reproduce

Steps to reproduce the behavior:

1. Download an AArch64 image from Hydra
2. Use etcher or dd to flash a micro sd card
3. Insert micro sd and boot a Pi Zero 2

Expected behavior

Expect it to boot to a nixOS shell

Workaround

It is possible to workaround the issue by mounting the micro SD on a operable Linux system by copying the RPi-3-b dtb file to the expected Pi Zero 2 file name:

cd /run/media/${USER}/NIXOS_SD/boot/nixos/gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom/
cp bcm2837-rpi-3-b.dtb bcm2837-rpi-zero-2.dtb

Then unmount the SD and boot with your RPi Zero 2

Notify maintainers

Notify the 2 folks listed as having touched the RPi Zero 2 dtb line in linux-rpi.nix
@leo60228
@zhaofengli

[zhaofengli]

Could you try the new_kernel variant of the SD image? It should have the DTB which was added in 5.18. The Hydra images use the mainline kernel, and linux-rpi.nix carries the RPi Foundation kernel. Alternatively, you can try this expression if you want to use the Foundation kernel.

[deftdawg]

Thanks @zhaofengli!

Tried the new_kernel variant... still panics because of a typo.

The file expected by uboot is

/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom/bcm2837-rpi-zero-2.dtb

The file provided by the image has -w in it's name:

/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom/bcm2837-rpi-zero-2-w.dtb

Copying the file:

cd /run/media/${USER}/NIXOS_SD/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom
sudo cp bcm2837-rpi-zero-2-w.dtb bcm2837-rpi-zero-2.dtb

Allows it to boot 👍 and wlan0 shows up (unlike with the RPi3 workaround above)... sweet!

[deftdawg]

Looks like I can get things to run, though the Pi Zero 2 needs help to keep from dying when running nixos-rebuild switch as it runs out of memory.

I've made a couple of tweaks:

1. Disable RPi Camera support and reduce VRAM to 16MB from the default of 128MB.

   edit /run/media/${USER}/FIRMWARE/config.txt:

   [pi02]
   kernel=u-boot-rpi3.bin
   # Give up VRAM for more Free System Memory
   # - Disable camera which automatically reserves 128MB VRAM
   start_x=0 
   # - Reduce allocation of VRAM to 16MB minimum for non-rotated (32MB for rotated)
   gpu_mem=16

2. Create a 2 GB swap file in root

   export SWAP_FILE=/swap
   # create a 2GB swap file
   if [ ! -f ${SWAP_FILE} ]; then
     sudo dd if=/dev/zero of=${SWAP_FILE} bs=100M count=20
     # format file as swap fs
     sudo mkswap -f ${SWAP_FILE}
   fi
   # activate swap file
   sudo swapon ${SWAP_FILE}
   echo

Still need to figure out permanent configuration, but at least with these I can rebuild without killing the Pi.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/raspberry-pi-compute-module-3-compatibility/16210/11

[deftdawg]

Looks like the problem is stale references in the uboot bins

[deftdawg@nixos:~/sda1]$ grep  bcm2837-rpi-zero-2.dtb *                                   
grep: u-boot-rpi3.bin: binary file matches
grep: u-boot-rpi4.bin: binary file matches

[deftdawg@nixos:~/sda1]$ strings u-boot-rpi3.bin | grep broadcom
... 
broadcom/bcm2837-rpi-zero-2.dtb
... 

This is based on the last new kernel sd image on hydra from Mar 20, 2023.

[deftdawg]

Looks like the problem was fixed in uboot 17 days ago... So we'd need a uboot update in nixos and a working hydra build

u-boot/u-boot@0e8c940

[plmercereau]

The PR with a newer version of uboot has been merged to nixpkgs, but it unfortunately does not solve the problem for me.

I created a small module that copies the right uboot file, creates a swap partition and allows to add options to the firmware configuration. Here's the gist.

Eventually my config looks like this:

{ config, lib, pkgs, ... }:
{
  imports = [
    <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64-installer.nix>
    ./sd-image.nix
  ];
  system.stateVersion = "23.11";

  # Pi Zero 2 struggles to work without swap
  sdImage.swap.enable = true;
  sdImage.extraFirmwareConfig = {
    # Give up VRAM for more Free System Memory
    # - Disable camera which automatically reserves 128MB VRAM
    start_x = 0;
    # - Reduce allocation of VRAM to 16MB minimum for non-rotated (32MB for rotated)
    gpu_mem = 16;
  };
  # bzip2 compression takes loads of time with emulation, skip it. Enable this if you're low on space.
  sdImage.compressImage = false;

  networking = {
    interfaces."wlan0".useDHCP = true;
    wireless = {
      enable = true;
      interfaces = [ "wlan0" ];
      networks = {
        "<ssid>" = {
          psk = "<ssid-key>";
        };
      };
    };
  };

  # Enable OpenSSH out of the box.
  services.sshd.enable = true;

  # NTP time sync.
  services.timesyncd.enable = true;

}

[alex-nt]

@plmercereau many thanks, I used your config and added it to my fork of nixpkgs here: https://github.com/NixOS/nixpkgs/compare/release-23.05...alex-nt:nixpkgs:feature-23.05-rpi?expand=1

Using sd-image-aarch64-new-kernel-no-zfs-installer.nix everything works! But whenever I try the rpi kernel via

boot = {
    kernelPackages = pkgs.linuxPackages_rpi3;
}

I loose wifi (wlan0 fails to init). I've been using a fork of nixpkgs since 2022-12 till now on RPI02w, latest kernel always provided usable results compared with the rpi provided one. I wonder what I keep messing up.

[deftdawg]

For wifi I observed everytime I got a message in journalctl -b -u dhcpcd relating to expiring ipv6 my wifi would go down... not sure if what you've got is related, but this workaround did the trick for me:

#87802 (comment)

[plmercereau]

my wifi works fine with the above configuration (connecting to a router with DHCP enabled), but sometimes IPv6 somehow takes precedence over IPv4, for instance:

[nixos@nixos:~]$ ping www.github.com
PING www.github.com (140.82.121.4) 56(84) bytes of data.
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=1 ttl=49 time=38.9 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=2 ttl=49 time=31.6 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=3 ttl=49 time=28.8 ms
^C
--- www.github.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 28.768/33.080/38.882/4.261 ms


[nixos@nixos:~]$ ping www.google.com
PING www.google.com(ams16s32-in-x04.1e100.net (2a00:1450:400e:80c::2004)) 56 data bytes
^C
--- www.google.com ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8170ms


[nixos@nixos:~]$ ping -4 www.google.com
PING  (142.250.179.164) 56(84) bytes of data.
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=1 ttl=56 time=28.3 ms
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=2 ttl=56 time=27.6 ms
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=3 ttl=56 time=23.2 ms
^C
---  ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 23.231/26.375/28.291/2.240 ms

I don't know if it is related to your problem though. I will try your suggestion @deftdawg

[alex-nt]

@plmercereau @deftdawg

With the following?

boot = {
    kernelPackages = pkgs.linuxPackages_rpi3;
}

Guys, you are giving me hope here, I might burn a few hours tomorrow and check to see what happens. All in all I am still very happy with your config @plmercereau. I have everything working (even i2c, but I didn't get to test spi yet, last time I failed to get spi working). Thanks yet again!

[rjpcasalino]

I am to understand that the core issue pi-zero-2.dtb file missing isn't fixed with latest u-boot? Many thanks to @deftdawg for helping me boot my Zero 2 with that hint. I think we also need to persevere @plmercereau's config someplace to help others that might not be looking at GitHub issues and considering the u-boot update didn't fix this yet. Perhaps updating the NixOS ARM wiki? In any case, thanks all!

[rjpcasalino]

@deftdawg cp bcm2837-rpi-3-b.dtb bcm2837-rpi-zero-2.dtb is still the work around I use on a fresh pi zero 2 but @plmercereau would be nice to get your module to work on a flake system like it's done here for a Pi 3 https://myme.no/posts/2022-12-01-nixos-on-raspberrypi.html. The price of zero 2 has dropped and it would be nice to get a more stable system

[plmercereau]

Hello folks,
I'll try to write a flake example in the upcoming days, and adapt the gist accordingly.
As regards pushing this upstream, I don't think it is a good idea because:

• on second thought, creating a swap partition may not be a good idea, as it would tend to dramatically reduce the lifetime of SD cards (too many write operations). Maybe we should try an option in using zram swap, or to suggest building the system remotely instead of on the pi zero
• I feel the cp bcm2837-rpi-3-b.dtb bcm2837-rpi-zero-2.dtb line is too hacky and should be solved somewhere else - but where? :/

[rjpcasalino]

awesome! looking forward to it. I tried a bit but couldn't manage to get anything working since I've moved to fully using flakes. Many thanks. I'm not sure where cp bcm2837-rpi-3-b.dtb bcm2837-rpi-zero-2.dtb should go either... @zhaofengli sorry to ping but thought you might be able to help figure that one out? The latest .img doesn't work without this hack but it would be nice if we could "support" the zero 2 a bit better but I'm clueless here...

[plmercereau]

Hello,
I wrote a small example that works on my end with flakes: https://github.com/plmercereau/nixos-pi-zero-2
What it does (and doesn't):

• when using the unstable channel, there is no need to use the cp bcm2837 ... thing. It works as is, as long as the add these lines for enabling the wifi:

  hardware.enableRedistributableFirmware = lib.mkForce false;
  hardware.firmware = [pkgs.raspberrypiWirelessFirmware];

• for the swap issue, I preferred the option to build the system in another machine. I used deploy-rs, but I am sure it would work with nixos-rebuild --target-host too
• Still couldn't find a way to use boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi3...

[rjpcasalino]

Nice! The flake works like a dream!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-on-raspberry-pi-zero-w/38018/1

[deftdawg]

Been a minute since I had a chance to play with this, but the hydra images @ https://hydra.nixos.org/job/nixos/release-23.11/nixos.sd_image_new_kernel_no_zfs.aarch64-linux now boot on my Pi Zero 2W without needing to copy dtb files, though I'm missing or forgetting a step as plugging an OTG adapter isn't working with my keyboard.

One thing to note the no_zfs work, however the other build https://hydra.nixos.org/job/nixos/release-23.11/nixos.sd_image.aarch64-linux, which I presume use zfs, don't work on my pi.

Going to close this since it seems the original defect is now solved. 🍻