Boot-loader nixos-rebuild switch with unexpected behavior

[johnalotoski]

Describe the bug
If switching between boot loaders in the nixos configuration, such as from grub efi to systemd-boot, a nixos-rebuild switch will not give an error but the boot loader change isn't carried out either and new system generations are not subsequently booted into.

This was initially encountered in nixops deployments where a bootloader change had occurred in the code and where nixops reported successful deploys, that a reboot was required, and the reboot did not result in system generation update.

To Reproduce
The easiest way may be using @cleverca22's nix-tests repo:

git clone https://github.com/cleverca22/nix-tests.git
cd nix-tests

# Run the following build to create a qemu/qcow virtual machine to test
nix-build simple-test.nix -A nvme

# Start the qemu virtual machine with the first result script:
result/bin/qemu_test1

# When the qemu system is available, type the following within the qemu machine console to install NixOS on grub EFI:
justdoit

# When the NixOS grub EFI install is done from within the qemu virtual machine, shut it down
shutdown

# Boot the qemu virtual machine from the installed grub EFI boot device:
result/bin/qemu_test2

# From within the vm, modify the following files to switch to systemd-boot:
# /etc/nixos/configuration.nix and /etc/nixos/generated.nix

# Comment out lines:
# boot.loader.grub.enable = true;
# boot.loader.grub.version = 2;
# boot.loader.grub.efiInstallAsRemovable = true;
# boot.loader.grub.efiSupport = true;
# boot.loader.grub.device = "nodev";

# Add lines to generated.nix:
boot.loader = {
  systemd-boot.enable = true;
  efi.canTouchEfiVariables = true;
}

# Rebuild to the new configuration
# No errors are shown and the activation appears to complete successfully:
nixos-rebuild switch

building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/<snip>-systemd-boot-builder.py.drv
  /nix/store/<snip>-nixos-system-nixos-<snip>.drv
building '/nix/store/<snip>-systemd-boot-builder.py.drv'
building '/nix/store/<snip>-nixos-system-nixos-<snip>.drv'
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles

# Reboot into what should be the new systemd-boot loader, 
# but observe Grub with only 1 generation
systemctl reboot

# From the qemu vm console, change to EF00 paritition type for the /boot mount
# Required when using the --install-bootloader nixos-rebuild switch option to go to systemd-boot
# Otherwise an error about a `wrong type for an efi system partition (ESP)` is shown
nix-env -iA nixos.gptfdisk
gdisk /dev/nvme0n1
t
1
EF00
w

# Rebuild to the new configuration using the additional --install-bootloader flag
# No errors are shown and the activation appears to complete successfully.
# This time systemd-boot is installed successfully
nixos-rebuild switch --install-bootloader

building Nix...
building the system configuration...
Created "/boot/EFI/systemd"
<...snip...>
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles

# Reboot into new systemd-boot
systemctl reboot

Expected behavior
Consider changing to behavior of: If a boot loader change is done in the nixos configuration and --install-bootloader is not specified for the rebuild, throw an error instead of appearing to succeed but failing to carry out the boot loader config change.

Additional context
There may be additional edge cases to consider when switching boot loaders, such as if going from grub-efi to systemd-boot and back to grub-efi. When doing so, if the state file for grub is still present at /boot/grub/state from the initial boot loader deployment, will grub re-install itself on the switch back?

Metadata
From the qemu vm example above:

[root@nixos:~]# nix run nixpkgs.nix-info -c nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.67, NixOS, 19.03.173322.67135fbcc5d (Koi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2.2`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

module:
https://github.com/NixOS/nixpkgs/tree/master/nixos/modules/system/boot/loader

cc: @cleverca22 @disassembler

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.