generate-zbm EFI bundle doesn't kexec into kernel, prebuilt EFI kexecs fine

[ahesford]

Discussed in #598

^{Originally posted by Contik March 1, 2024}
Hi all,

I have an Arch Linux system with ZFSBootMenu running. It works perfectly fine with the prebuilt EFI release image downloadable at https://get.zfsbootmenu.org/efi. However, I can't get a generate-zbm EFI bundle to kexec into my kernel and initramfs; only the prebuilt image can do that.

When I boot into my self-compiled EFI bundle ZFSBootMenu doesn't seem to correctly hand control over to the ZFS dataset's kernel and initramfs and instead reboots the machine. With the prebuilt EFI release image no such reboot happens, that one just works perfectly.

I guess this post boils down to: what makes the prebuilt EFI release image kexec into my kernel and initramfs that I can't replicate with the self-built EFI bundle? I'm assuming kernel and initramfs files are perfectly fine since they do work when paired with the prebuilt EFI release image. Even my self-compiled image can see my kernel and initramfs, it just doesn't boot into them. Is there some debugging I can do to better understand what exactly is causing a reboot where the prebuilt EFI release image instead does a clean kexec?

I'd appreciate a second pair of eyes pointing me at what I'm missing. Thanks!

Here's what I have:

• Arch Linux with latest zfsbootmenu package (releases/tag/v2.3.0 from this repo)
• initramfs gets created via mkinitcpio with dependencies met per docs:
  
• Disk is GPT-partitioned
• Dual boot Windows/Linux setup
• File /etc/zfsbootmenu/config.yaml has:

  Global:
    ManageImages: true
    BootMountPoint: /efi
    InitCPIO: true
  Components:
    Enabled: false
  EFI:
    ImageDir: /efi/EFI/ZBM
    Versions: false
    Enabled: true
  Kernel:
    CommandLine: ro loglevel=0 zbm.import_policy=hostid
    Prefix: vmlinuz
  

• File /etc/zfsbootmenu/mkinitcpio.conf has:

  # sed -e '/^#/d' -e '/^$/d' /etc/zfsbootmenu/mkinitcpio.conf
  MODULES=()
  BINARIES=()
  FILES=()
  HOOKS=(base udev autodetect modconf block filesystems keyboard)
  

• EFI System Partition is mounted at /efi where /efi/EFI/ZBM/vmlinuz.EFI sits next to Windows Boot Manager files:

  /efi
  ├── EFI
  │   ├── Boot
  │   │   └── ...
  │   ├── Microsoft
  │   │   └── ...
  │   └── ZBM
  │       └── vmlinuz.EFI
  ├── ...
  └── ...
  

• The one and only existing mkinitcpio preset file at /etc/mkinitcpio.d/linux.preset has:

  ALL_kver="/boot/vmlinuz-linux"
  ALL_microcode=(/boot/*-ucode.img)
  PRESETS=('default')
  default_image="/boot/initramfs-linux.img"
  

• File /etc/mkinitcpio.conf has:

  # sed -e '/^#/d' -e '/^$/d' /etc/mkinitcpio.conf
  MODULES=()
  BINARIES=()
  FILES=(/etc/zfs/zpool.key)
  HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont numlock block zfs filesystems)
  

• ZFS zpool has bootfs property set:

  # zpool get bootfs
  NAME   PROPERTY  VALUE                     SOURCE
  zpool  bootfs    zpool/root/archlinux-frn  local
  

• Where dataset zpool/root/archlinux-frn is currently the only root boot environment
• Boot environment is encrypted and has org.zfsbootmenu:commandline set like so:

  # zfs get encryptionroot,keylocation,encryption,org.zfsbootmenu:commandline zpool/root/archlinux-frn
  NAME                      PROPERTY                     VALUE                                       SOURCE
  zpool/root/archlinux-frn  encryptionroot               zpool                                       -
  zpool/root/archlinux-frn  keylocation                  none                                        default
  zpool/root/archlinux-frn  encryption                   aes-256-gcm                                 -
  zpool/root/archlinux-frn  org.zfsbootmenu:commandline  rw nowatchdog rd.vconsole.keymap=de-latin1  local
  

• The keylocation in my encryptionroot is set as:

  # zfs get keylocation zpool
  NAME   PROPERTY     VALUE                      SOURCE
  zpool  keylocation  file:///etc/zfs/zpool.key  local
  

• Where mkinitcpio -P will embed the same zpool.key file into initramfs via /etc/mkinitcpio.conf
• One plain regular linux kernel and an initramfs live at /boot inside the encrypted zpool

  /boot
  ├── amd-ucode.img
  ├── initramfs-linux.img
  └── vmlinuz-linux
  

• lsinitcpio confirms that the key file lands in my initramfs:

  # lsinitcpio /boot/initramfs-linux.img | grep 'etc/zfs/zpool.key'
  etc/zfs/zpool.key
  ```</div>
  

[ahesford]

I've confirmed that this is an upstream kernel problem, and fixes are out of our hands. It will be fixed in Void (and any future ZBM build containers) once void-linux/void-packages#49529 lands.