Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos: Formally deprecate boot.loader.raspberryPi #241534

Conversation

samueldr
Copy link
Member

@samueldr samueldr commented Jul 4, 2023

The whole option set was recommended against since mid-2019, and never worked with the Raspberry Pi 4 family of devices.

We should have deprecated it in early 2020 for removal by 2021. At the time I did not feel confident in making such a decision, and never ended-up getting around to it.

The only supported-by-NixOS boot methods for AArch64 are standards-based boot methods, namely UEFI or the pragmatically almost-standard extlinux-compatible for U-Boot.

You can quote me on that.

Description of changes

Added a deprecation warning, and added note and schedule for deprecation in release notes.

Things done

Ran:

.../nixos-unstable $ nix-instantiate ./nixos/release.nix -A sd_image.aarch64-linux
trace: warning: system.stateVersion is not set, defaulting to 23.11. Read why this matters on https://nixos.org/manual/nixos/stable/options.html#opt-system.stateVersion.
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/aaj644zszsm1gkza613ja9z01qz1yqwz-nixos-sd-image-23.11pre130979.gfedcba-aarch64-linux.img.drv

Also built documentation, and checked the result is as expected for options

.../nixos-unstable $ nix-build ./nixos/release.nix -A manualHTML.x86_64-linux

As expected, no warnings.

Was not tested: Using those options, as I have no system that are using those options. I know it sucks sending "untested" changes, but I'm at least honest with what I have done.


  • Built on platform(s)
    • N/A only docs and NixOS evaluation affected
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • Notable change / incompatibiliy
  • Fits CONTRIBUTING.md.

The whole option set was recommended against since mid-2019, and never
worked with the Raspberry Pi 4 family of devices.

We should have deprecated it in early 2020 for removal by 2021. At the
time I did not feel confident in making such a decision, and never
ended-up getting around to it.

The ***only*** supported-by-NixOS boot methods for AArch64 are
standards-based boot methods, namely UEFI or the pragmatically
almost-standard extlinux-compatible for U-Boot.

You can quote me on that.
@bct
Copy link
Contributor

bct commented Jul 6, 2023

👍 These clear deprecation notices would have saved me a bunch of time spent going down blind alleys.

Copy link
Member

@RaitoBezarius RaitoBezarius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Beautiful.

Copy link
Contributor

@K900 K900 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kill it. With fire.

@RaitoBezarius RaitoBezarius merged commit 828633d into NixOS:master Jul 8, 2023
24 checks passed
@samueldr samueldr deleted the fix/finally-deprecate-boot-loader-raspberrypi branch July 13, 2023 01:44
@SFrijters
Copy link
Member

SFrijters commented Jul 16, 2023

@samueldr Sorry, I have a question: I don't know much about the details of these parameters, but I do use boot.loader.raspberryPi.firmwareConfig to append some options to /boot/config.txt (which indeed appends these lines, and the lines do actually do something). What would be the new way to do this? This is on a Raspberry Pi 4 by the way.

@hannesg
Copy link

hannesg commented Jul 30, 2023

@samueldr I also have some questions. The NixOs Wiki doesn't recommend against the raspberry bootloader at all. See: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi . Much to the contrary, some documentation in that article seems to me to only work correctly when using that bootloader. My installation is also just a year old and I did not even know that I did something unsupported until today. Can you update the Wiki with an official recommendation or upgrade path and also add a link to the error message with information regarding this deprecation?

@samueldr
Copy link
Member Author

Thanks for noticing this discrepancy.

The NixOS wiki instructions on that page are from contributions written before the change, and have never been updated to remove those recommendations.

They do not work as expected. The NixOS wiki pages for the Raspberry Pi are outdated.

Note that the "current" Raspberry Pis have their own pages:

(If anyone else wanted to remove this all and make a note.)

I'll try going back to this, but I've been relatively busy recently.

@hannesg
Copy link

hannesg commented Aug 4, 2023

@samueldr , thank you for the heads up. I left a warning on this page about the deprecation: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi

@SFrijters
Copy link
Member

@samueldr I noticed this deprecation warning again today while doing some maintenance on my Pi and I would still appreciate some pointers on what the migration path for boot.loader.raspberryPi.firmwareConfig is. I use it to set a bunch of dtparam values to switch off various LEDs:

        firmwareConfig = ''
          dtparam=act_led_trigger=none
          dtparam=act_led_activelow=off
          dtparam=pwr_led_trigger=default-on
          dtparam=pwr_led_activelow=off
          dtparam=eth_led0=4
          dtparam=eth_led1=4
        '';

@K900
Copy link
Contributor

K900 commented Sep 23, 2023

Device tree overlays, probably.

@SFrijters
Copy link
Member

I've tried to migrate from boot.loader.raspberryPi to boot.loader.generic-extlinux-compatible (via the Raspberry Pi 4 module in nixos-hardware). I think I've got the dtparams -> device tree overlay solved, but it looks like the system keeps booting into the last generation that uses boot.loader.raspberryPi; what is the migration path here? I do see new entries in /boot/extlinux/extlinux.conf, and also the old ones from a couple of days ago before I made the aforementioned changes. This is all a bit of a pain.

@sorki
Copy link
Member

sorki commented Oct 16, 2023

I've tried to migrate from boot.loader.raspberryPi to boot.loader.generic-extlinux-compatible (via the Raspberry Pi 4 module in nixos-hardware). I think I've got the dtparams -> device tree overlay solved, but it looks like the system keeps booting into the last generation that uses boot.loader.raspberryPi; what is the migration path here? I do see new entries in /boot/extlinux/extlinux.conf, and also the old ones from a couple of days ago before I made the aforementioned changes. This is all a bit of a pain.

On rPi, the actual bootloader is located on the first SD card partition. generic-extlinux only generates /boot/extlinux/extlinux.conf which is loaded by bootloader on /dev/mmcblk0p1 that is typically not mounted and even if it was we don't have an update mechanism for its contents (i.e. no automatic updates of /boot/firmware).

You have several options how to fix this

The last one would be really neat.

If sd-image<...>.nix is imported you have access to config.sdImage.populateFirmwareCommands but it isn't a derivation, it looks like it could be turned into one and the image builder would just cp the output into place.

At that point you could just grab that using something like

nix-build '<nixpkgs/nixos>' -A config.sdImage.populateFirmwareCommands -I nixos-config=..

and it could also be used by the update mechanism. I can help with all this I think.

Edit: this was actually previously done by https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/loader/raspberrypi/raspberrypi.nix#L146

@SFrijters
Copy link
Member

Thanks, that's very helpful.

I'm actually booting from a SSD drive via USB (and using a ZFS / ephemeral setup as well), so it looks a bit different for me:

$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda      8:0    0 223.6G  0 disk 
├─sda1   8:1    0  1023M  0 part /boot
├─sda2   8:2    0     8G  0 part [SWAP]
└─sda3   8:3    0 214.6G  0 part 
zram0  253:0    0   1.8G  0 disk [SWAP]

But, looking at the nix files you linked, it looks like the stuff from the 'old' boot setup is sitting in my /boot/ (which is defined as a fileSystems entry in hardware-configuration.nix):

ls /boot
bcm2710-rpi-2-b.dtb       bcm2710-rpi-zero-2.dtb    bcm2711-rpi-cm4.dtb       bcm2837-rpi-3-b.dtb       bcm2838-rpi-4-b.dtb  config.txt    fixup4db.dat  fixup_db.dat  kernel.img      old           start4x.elf   start_x.elf
bcm2710-rpi-3-b.dtb       bcm2710-rpi-zero-2-w.dtb  bcm2711-rpi-cm4-io.dtb    bcm2837-rpi-3-b-plus.dtb  bootcode.bin         extlinux      fixup4x.dat   fixup_x.dat   kernel.img.bak  start4cd.elf  start_cd.elf
bcm2710-rpi-3-b-plus.dtb  bcm2711-rpi-400.dtb       bcm2711-rpi-cm4s.dtb      bcm2837-rpi-cm3.dtb       cmdline.txt          fixup4cd.dat  fixup_cd.dat  initrd        nixos           start4db.elf  start_db.elf
bcm2710-rpi-cm3.dtb       bcm2711-rpi-4-b.dtb       bcm2837-rpi-3-a-plus.dtb  bcm2837-rpi-zero-2.dtb    cmdline.txt.bak      fixup4.dat    fixup.dat     initrd.bak    nixos-init      start4.elf    start.elf

So I think if I do follow the script in https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/sd-card/sd-image-aarch64.nix#L22 I should just splat all these files into /boot (and update config.txt)?

@samueldr
Copy link
Member Author

On rPi, the actual bootloader is located on the first SD card partition. generic-extlinux only generates /boot/extlinux/extlinux.conf which is loaded by bootloader on /dev/mmcblk0p1 that is typically not mounted and even if it was we don't have an update mechanism for its contents (i.e. no automatic updates of /boot/firmware).

I'm not sure I understand this correctly, but it might be misleading.

  • generic-extlinux will generate the extlinux/extlinux.conf file on whichever filesystem /boot comes from. It does not care about the partition.
  • U-Boot, the platform firmware installed in the first partition for Raspberry Pi, does not care about the partition outright, as long as it's marked "bootable", either the (legacy) MBR flag, or marked as the ESP, it will load [/boot]/extlinux/extlinux.conf from the filesystem.

We have mechanisms to update /boot/extlinux/extlinux.conf.

The first partition shouldn't be mounted ever, unless you want to manually edit config.txt to change options, but even then it's a rare one-off for people who know they have to.

The migrations step will be varied from situation to situation due to the myriad ways to setup a system.


Looking at your lsblk output, you have an unsupported layout from the olden times, or a custom made install.

If you want to have the rootfs on a filesystem unsupported by U-Boot, you will need to at least have a /boot partition with a filesystem supported by U-Boot (FAT is okay, ext4 preferred if not doing "by-the-spec" UEFI boot).

@sorki
Copy link
Member

sorki commented Oct 16, 2023

I'm not sure I understand this correctly, but it might be misleading.

Oh, the mountpoint is defined but has 'noauto' and the directory is missing. Cool! You are right that this depends on the setup as well, I mostly use SD images.

Still the firmware update mechanism would be nice.

@sorki

This comment was marked as outdated.

@sorki

This comment was marked as outdated.

@SFrijters
Copy link
Member

SFrijters commented Oct 16, 2023

Trying to make my use case less vague: I set this up ~a year ago, so I wouldn't call it 'the olden times', but it is a custom install, based on https://grahamc.com/blog/erase-your-darlings/ , with added Raspberry flavour.

I have uploaded a gist with the (sanitized) version of my configuration and the custom scripts I used, along with the README I keep with it, and a (reduced) diff of my config: https://gist.github.com/SFrijters/665686199c915804f993a4ee4002d33e/ . I hope this retains enough info to be helpful but is small enough to not scare people off.

The firmware update mechanism @sorki mentioned is I think what I expected to happen when I removed the boot.loader.raspberryPi part of the configuration and set another bootloader (via the nixos-hardware configuration); the fact that the configuration built just fine but was not picked up was a big surprise for me (and because I was also rather unsure about the original device tree overlays it led me down the wrong path at first).

EDIT: Indeed I boot without SD card, after the initial setup as described in the gist. The system has been working fine, but I do recall it being a bit finicky with respect to actually booting from USB the first time, so I would prefer not to nuke my current setup, even with the supposed reproducibility of my scripts and config.

@sorki
Copy link
Member

sorki commented Oct 17, 2023

That's neat @SFrijters! Regarding act and power LED configuration you can try my module from https://github.com/pi-bouncer/pi-bouncer/blob/3fcf4b5b2f1bc28821a31ba8ada4a34f0c9d08bb/modules/rpi.nix (don't know how to do eth leds tho).

Thank you @samueldr for deprecating this, it will cause a lot of pain for users using it but it is the right thing to do long-term.

@SFrijters
Copy link
Member

@sorki Thanks! This is how I'm doing it now, all based on the device tree overlays - at least I'm fairly confident it will work once I can actually switch into the new boot generation! https://gist.github.com/SFrijters/206d2c09656affb04284f076c75a1969

@SFrijters
Copy link
Member

SFrijters commented Oct 22, 2023

All is well that ends well: I did not like having to do this migration (and I still think it would've been nice to get better documentation on a migration path as a part of this PR instead of just a "not like this" deprecation warning), but I really appreciate @sorki and @samueldr for their help.

Using #261857 to move to the new bootloader and the overlays in https://gist.github.com/SFrijters/206d2c09656affb04284f076c75a1969 to replace my dtparams I have now migrated succesfully while keeping my tweaks intact.

@dbarrosop
Copy link
Contributor

I just landed here due to the deprecation notice:

trace: warning: The option set for `boot.loader.raspberrypi` has been recommended against
for years, and is now formally deprecated.

If I am understanding the comments around here the solution isn't as simple as just as replacing the raspberry-related settings with:

boot.loader.generic-extlinux-compatible.enable = true;

What are the steps to migrate? @SFrijters I am also booting from SSD and I also remember a painful experience making it work, which steps did you follow to migrate? I don't have a need for device tree overlays though.

@SFrijters
Copy link
Member

SFrijters commented Nov 19, 2023

@dbarrosop: I ended up using https://github.com/NixOS/nixos-hardware (which sets boot.loader.generic-extlinux-compatible.enable = true;, and maybe does some other things by default, but probably setting it directly is enough - I did it because it had options related to the device tree overlays).

Then I used this procedure (taken from a comment in my config so I don't forget it myself):

Generating contents for /boot: #261857

 {
   imports = [
    ./modules/installer/sd-card/sd-image-aarch64.nix
   ];
   nixpkgs = {
     crossSystem.system = "aarch64-linux";
     overlays = [ ];
   };
 }
  • Build the contents of the firmware partition:
    $ nix-build . -A config.sdImage.firmwarePartitionContents -I nixos-config=./firmware-rpi4.nix
  • Copy the contents of ./result to the /boot partition of the Raspberry Pi (keep existing content in mind - maybe move them to a backup directory)
  • (Optional) remove unused/old files from /boot (e.g. ones that were not timestamped after the switch to boot.loader.generic-extlinux-compatible.enable = true;)
  • Remaining content:
    armstub8-gic.bin
    bcm2711-rpi-400.dtb
    bcm2711-rpi-4-b.dtb
    bcm2711-rpi-cm4.dtb
    bcm2711-rpi-cm4s.dtb
    bootcode.bin
    config.txt
    extlinux/extlinux.conf
    fixup4cd.dat
    fixup4.dat
    fixup4db.dat
    fixup4x.dat
    fixup_cd.dat
    fixup.dat
    fixup_db.dat
    fixup_x.dat
    nixos/<overlays, Image, initrd files>
    start4cd.elf
    start4db.elf
    start4.elf
    start4x.elf
    start_cd.elf
    start_db.elf
    start.elf
    start_x.elf
    u-boot-rpi3.bin
    u-boot-rpi4.bin
    

Hope this helps!

@dbarrosop
Copy link
Contributor

Yes, that's extremely helpful, and terrifying, thanks!

@newAM
Copy link
Member

newAM commented Nov 24, 2023

I noticed this deprecation warning again today while doing some maintenance on my Pi and I would still appreciate some pointers on what the migration path for boot.loader.raspberryPi.firmwareConfig is. I use it to set a bunch of dtparam values to switch off various LEDs:

        firmwareConfig = ''
          dtparam=act_led_trigger=none
          dtparam=act_led_activelow=off
          dtparam=pwr_led_trigger=default-on
          dtparam=pwr_led_activelow=off
          dtparam=eth_led0=4
          dtparam=eth_led1=4
        '';

I have the a similar question. Does anyone know what the replacement is for driving default GPIO values via boot.txt?

firmwareConfig = ''
    gpio=5,12,6,13,19,16=op,dl
'';

@sorki
Copy link
Member

sorki commented Nov 24, 2023

I noticed this deprecation warning again today while doing some maintenance on my Pi and I would still appreciate some pointers on what the migration path for boot.loader.raspberryPi.firmwareConfig is. I use it to set a bunch of dtparam values to switch off various LEDs:

        firmwareConfig = ''
          dtparam=act_led_trigger=none
          dtparam=act_led_activelow=off
          dtparam=pwr_led_trigger=default-on
          dtparam=pwr_led_activelow=off
          dtparam=eth_led0=4
          dtparam=eth_led1=4
        '';

I have the a similar question. Does anyone know what the replacement is for driving default GPIO values via boot.txt?

Sort-of already in this thread 😄

firmwareConfig = ''
    gpio=5,12,6,13,19,16=op,dl
'';

This should still work (i.e. if it's already in the config.txt, you can simply remove (comment out) the firmwareConfig from you configuration.nix, better to port it to sdImage.populateFirmwareCommands which #261857 turns into sdImage.firmwarePartitionContents so it can be built and updated manually.

@newAM
Copy link
Member

newAM commented Nov 24, 2023

Thanks for the pointers to sdImage.populateFirmwareCommands and sdImage.firmwarePartitionContents! That looks like exactly what I want!

I looked at the conversation about device trees, but I do not think that solves my problem since I need the GPIOs driven low very quickly (before the kernel starts) after boot to work around a hardware bug. My understanding of device trees is that they are all kernel-space and will not do anything to the low-level boot firmware (e.g. /boot/config.txt), is that correct?

@sorki
Copy link
Member

sorki commented Nov 24, 2023

I looked at the conversation about device trees, but I do not think that solves my problem since I need the GPIOs driven low very quickly (before the kernel starts) after boot to work around a hardware bug. My understanding of device trees is that they are all kernel-space and will not do anything to the low-level boot firmware (e.g. /boot/config.txt), is that correct?

Correct but the config.txt and the firmware is still there, it just loads uBoot which loads the device trees (so the fw cannot apply overlays on them and pass that to kernel). Most options of config.txt should still work.. I think 🙃

@axelsimon
Copy link

Then I used this procedure (taken from a comment in my config so I don't forget it myself):

(…)

Hope this helps!

It absolutely did, thank you @SFrijters.
I did end up with a few extra files after cleaning up old files. These have a recent timestamp, i'm not sure what they are¹, but sharing them in case someone else ends up in a similar situation:

-rwxr-xr-x 1 root root   80 Feb 14 15:06 pieeprom.sig
-rwxr-xr-x 1 root root 512K Feb 14 15:06 pieeprom.upd
-rwxr-xr-x 1 root root  86K Feb 14 15:06 recovery.bin
-rwxr-xr-x 1 root root  98K Feb 14 15:06 vl805.bin
-rwxr-xr-x 1 root root   80 Feb 14 15:06 vl805.sig

None of these are built by the config.sdImage.firmwarePartitionContents attribute, as they are not in my ./result directory.

  1. They might from an attempt at switching to systemd-boot, a few days ago.

kalbasit added a commit to kalbasit/soxincfg that referenced this pull request Mar 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet