Boot broken on Raspberry Pi CM4 when using linux_rpi4 kernel package

[n-hass]

Describe the bug

Booting a Raspberry Pi Compute Module 4 with boot.kernelPackages = pkgs.linuxPackages_rpi4 is not possible. Booting freezes at "Starting kernel ..." and does not progress, no other activity (network, any monitor, serial) can be detected.

This is caused by loading the dtb bcm2711-rpi-cm4.dtb. When using the mainline kernel, only the dtbs bcm2711-rpi-400.dtb bcm2711-rpi-4-b.dtb bcm2711-rpi-cm4-io.dtb get placed in the corresponding boot entry files in /boot/nixos directory.

The mainline kernel config still tries to look for bcm2711-rpi-cm4.dtb, but it does not find it and every boot you will get an error message pre-stage 1:

Retrieving file: /extlinux/../nixos/yrygi9cnd2fkjpxgj82s0s86wkc5fzzy-device-tree-overlays/broadcom/bcm2711-rpi-cm4.dtb
** File not found /extlinux/../nixos/yrygi9cnd2fkjpxgj82s0s86wkc5fzzy-device-tree-overlays/broadcom/bcm2711-rpi-cm4.dtb **
Moving Image from 0x80000 to 0x200000, end=3fe0000
## Flattened Device Tree blob at 2eff2900
   Booting using the fdt blob at 0x2eff2900
Working FDT set to 2eff2900
   Using Device Tree in place at 000000002eff2900, end 000000002f002f10
Working FDT set to 2eff2900

but it continues to boot into stage 1 and so on.

Notably copying the missing dtb to the location that the mainline kernel initrd is looking for so it can load it, causes the same boot failure.

Steps To Reproduce

Steps to reproduce the behavior:

1. Flash the CM4 with sd-image 23.05.4930.f741f8a83991-aarch64-linux or 23.11beta509.21af6ad5395b-aarch64-linux
2. Add boot.kernelPackages = pkgs.linuxPackages_rpi4 to configuration.
3. nixos-rebuild and reboot

Expected behavior

Boot should progress to login screen.

Additional context

A custom carrier board is being used from Seeed Studio for their EdgeBox RPI-200. Behaviour is reproducible on a standard CM4 IO board though

stdout while booting with linux_rpi4 kernel:

Retrieving file: /extlinux/../nixos/fddn8wnkxv50xxq4s1f79jaakg72r8bd-linux-6.1.21-1.20230405-Image
Retrieving file: /extlinux/../nixos/1982hspygiq06mdlh7ndl7h8sz88h455-initrd-linux-6.1.21-1.20230405-initrd
append: init=/nix/store/dl4812xjijbnw0zdqr0fys8r7gpd2cim-nixos-system-logr-edge-nixos-base-23.11.1697.781e2a9797ec/init console=ttyS0,115200n8 loglevel=4
Retrieving file: /extlinux/../nixos/4jajp414rjdphms0nz9k3dg176wkc6a3-device-tree-overlays/broadcom/bcm2711-rpi-cm4.dtb
Moving Image from 0x80000 to 0x200000, end=20b0000
## Flattened Device Tree blob at 04700000
   Booting using the fdt blob at 0x4700000
Working FDT set to 4700000
   Using Device Tree in place at 0000000004700000, end 000000000471019a
Working FDT set to 4700000

Starting kernel …

Notify maintainers

@TredwellGit
@Ma27
@NeQuissimus
@alyssais
@thoughtpolice

Metadata

(This is running WITHOUT the kernelPackage as its not possible to boot with it)

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.1.63, NixOS, 23.11 (Tapir), 23.11beta509.21af6ad5395b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixos-23.11beta509.21af6ad5395b"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Priorities

Add a 👍 reaction to issues you find important.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pi4-broken-on-23-11-uefi-zfs-root/36375/2

[n-hass]

Please see my edits in the bug description.

The root cause is actually loading the bcm2711-rpi-cm4.dtb, not the patched kernel… mainline does not load this dtb, linux_rpi4 does, hence the issue is reproduced only with linux_rpi4. Removing the dtb does not fix the issue. loading the dtb in mainline reproduces the issue.

Could I please have some input on this problem?

[Majiir]

I've used linux_rpi4 on a CM4 for more than a year, and never had boot issues. I do have bcm2711-rpi-cm4.dtb in my dtbs folder.

Is your CM4's bootloader EEPROM updated? You might also want to check your firmware partition's config.txt (although the one included with the NixOS SD image has worked for me).