ZFS mirror installation instructions (or script)

[Jip-Hop]

Hi! My goal is to install Debian 11 Bullseye with a mirrored unencrypted ZFS root and zfsbootmenu to roll back snapshots in case the system breaks after an update.

Currently the wiki has steps to install without encryption, but doesn't mention the steps to install to multiple disks (ZFS mirror).

Does zfsbootmenu support installations with mirrored disks? If so, what are the steps to perform such an installation? :)

I found a couple of scripts which seem to greatly simplify the process of installing Debian with a mirrored ZFS root, however they don't include zfsbootmenu unfortunately.

• zfs-installer
• debian-buster-zfs-root

[ahesford]

ZFSBootMenu doesn't care how your pool is constructed.

When creating the pool as described in the guide, rather than specifying the single vdev $TARGET_VDEV, specify a mirror vdev:

zpool create -f [options ...] \    
      zroot mirror $TARGET_VDEV_1 $TARGET_VDEV_2

[ahesford]

ZFSBootMenu will boot your system as long as it can import your pool, which shouldn't be a problem if one of the disks fails.

The generate-zbm script will not handle copying the image to multiple EFI system partitions. If you want redundancy in the bootloader, you have a few of options:

1. Manually copy the ZBM images to an EFI system partition on your second disk. This isn't too bad because you don't need to recreate ZBM images frequently.
2. Write a post-run hook (see the PostHooksDir description in the generate-zbm man page) that will sync your two EFI system partitions every time you generate a new image.
3. Make your EFI system partitions an mdraid mirror (must be metadata v1.0 or lower for this to work).

You can also just ignore redundancy, because it's easy to dump a release image of ZBM on a USB key and boot from that if your regular boot disk fails.

[ahesford]

It turns out we already have a sample implementation of Option 2:

https://github.com/zbm-dev/zfsbootmenu/blob/master/contrib/esp-sync.sh

[Jip-Hop]

Thanks for the hints! I tried to follow the wiki steps + your suggestions in VirtualBox, but didn't succeed so far. First snag I hit is that I also had to install dosfstools, otherwise mkfs would fail. Rest of the steps seemed to complete without problems (except for the cryptsetup ERROR when running update-initramfs -c -k all... which I suppose I can ignore since I didn't setup encryption). But the installation doesn't want to boot...

Any idea where it went wrong?

vboxmanage list vms
# Inside VM become root with `sudo su`.
# Then edit /etc/apt/sources.list, should look like this:
# deb http://deb.debian.org/debian bullseye main contrib
# deb-src http://deb.debian.org/debian bullseye main contrib
# Use `vboxmanage controlvm DebianZFS keyboardputstring` to send keystrokes to the VM

# NOTE: also install `dosfstools`, was missing from wiki instructions
vboxmanage controlvm DebianZFS keyboardputstring 'apt install --yes debootstrap gdisk parted dkms linux-headers-$(uname -r) dosfstools'
vboxmanage controlvm DebianZFS keyboardputstring 'apt install --yes zfsutils-linux'

vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk --zap-all /dev/sda'
vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk -n1:1m:+512m -t1:ef00 /dev/sda'
vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk -n2:0:0 -t2:bf00 /dev/sda'

vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk --zap-all /dev/sdb'
vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk -n1:1m:+512m -t1:ef00 /dev/sdb'
vboxmanage controlvm DebianZFS keyboardputstring 'sgdisk -n2:0:0 -t2:bf00 /dev/sdb'

vboxmanage controlvm DebianZFS keyboardputstring 'lsblk'

vboxmanage controlvm DebianZFS keyboardputstring 'mkfs -t vfat -F 32 -s 1 -n EFI /dev/sda1'
vboxmanage controlvm DebianZFS keyboardputstring 'mkfs -t vfat -F 32 -s 1 -n EFI /dev/sdb1'

# Replace $TARGET_VDEV_1 $TARGET_VDEV_2 with persistent disk names (/dev/disk/by-id/ata-VBOX_HARDDISK_xxxxxx-xxxxxx-part2)
vboxmanage controlvm DebianZFS keyboardputstring 'zpool create -f -o ashift=12 -o autotrim=on -O acltype=posixacl -O compression=lz4 -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=none -R /mnt zroot mirror $TARGET_VDEV_1 $TARGET_VDEV_2'

vboxmanage controlvm DebianZFS keyboardputstring 'zfs create zroot/ROOT && zfs create -o canmount=noauto -o mountpoint=/ zroot/ROOT/debian && zfs mount zroot/ROOT/debian && zfs create -o mountpoint=/home zroot/home'
vboxmanage controlvm DebianZFS keyboardputstring 'zpool set bootfs=zroot/ROOT/debian zroot'
vboxmanage controlvm DebianZFS keyboardputstring 'zpool export zroot'
vboxmanage controlvm DebianZFS keyboardputstring 'zpool import -N -R /mnt zroot'
vboxmanage controlvm DebianZFS keyboardputstring 'zfs mount zroot/ROOT/debian'
vboxmanage controlvm DebianZFS keyboardputstring 'zfs mount -a'

vboxmanage controlvm DebianZFS keyboardputstring 'mkdir -p /mnt/boot/efi && mount /dev/sda1 /mnt/boot/efi'

vboxmanage controlvm DebianZFS keyboardputstring 'debootstrap bullseye /mnt'

vboxmanage controlvm DebianZFS keyboardputstring 'cp /etc/resolv.conf /mnt/etc/ && echo "debian" > /mnt/etc/hostname && echo -e "127.0.1.1\tdebian" >> /mnt/etc/hosts'
vboxmanage controlvm DebianZFS keyboardputstring 'for i in dev sys proc run; do mount --rbind /$i /mnt/$i && mount --make-rslave /mnt/$i; done'
vboxmanage controlvm DebianZFS keyboardputstring 'chroot /mnt /bin/bash --login'

vboxmanage controlvm DebianZFS keyboardputstring 'passwd'

vboxmanage controlvm DebianZFS keyboardputstring 'echo "deb http://deb.debian.org/debian bullseye main contrib" > /etc/apt/sources.list && echo "deb-src http://deb.debian.org/debian bullseye main contrib" >> /etc/apt/sources.list && echo "" >> /etc/apt/sources.list && echo "deb http://deb.debian.org/debian-security/ bullseye-security main contrib" >> /etc/apt/sources.list && echo "deb-src http://deb.debian.org/debian-security/ bullseye-security main contrib" >> /etc/apt/sources.list && echo "" >> /etc/apt/sources.list && echo "deb http://deb.debian.org/debian bullseye-updates main contrib" >> /etc/apt/sources.list && echo "deb-src http://deb.debian.org/debian bullseye-updates main contrib" >> /etc/apt/sources.list && echo "" >> /etc/apt/sources.list && echo "deb http://deb.debian.org/debian bullseye-backports main contrib" >> /etc/apt/sources.list && echo "deb-src http://deb.debian.org/debian bullseye-backports main contrib" >> /etc/apt/sources.list'

vboxmanage controlvm DebianZFS keyboardputstring 'apt update && apt full-upgrade && apt install --yes locales console-setup bash-completion'
vboxmanage controlvm DebianZFS keyboardputstring '. /usr/share/bash-completion/bash_completion'

vboxmanage controlvm DebianZFS keyboardputstring 'dpkg-reconfigure locales tzdata keyboard-configuration console-setup'

vboxmanage controlvm DebianZFS keyboardputstring 'apt install --yes linux-headers-amd64 linux-image-amd64 refind git zfs-initramfs'

vboxmanage controlvm DebianZFS keyboardputstring 'echo REMAKE_INITRD=yes | tee /etc/dkms/zfs.conf'
vboxmanage controlvm DebianZFS keyboardputstring 'zpool set cachefile=/etc/zfs/zpool.cache zroot'
vboxmanage controlvm DebianZFS keyboardputstring 'systemctl enable zfs.target && systemctl enable zfs-import-cache && systemctl enable zfs-mount && systemctl enable zfs-import.target'


vboxmanage controlvm DebianZFS keyboardputstring 'echo "$(blkid -o export /dev/sda1 | grep PARTUUID) /boot/efi vfat defaults 0 1" > /etc/fstab'

vboxmanage controlvm DebianZFS keyboardputstring 'zfs set org.zfsbootmenu:commandline="quiet" zroot/ROOT'

vboxmanage controlvm DebianZFS keyboardputstring 'mkdir -p /usr/local/src && cd /usr/local/src && git clone "https://github.com/zbm-dev/zfsbootmenu.git" && cd zfsbootmenu && make install'

vboxmanage controlvm DebianZFS keyboardputstring 'sed -i -e "s|void|debian|" -e "s|ManageImages: false|ManageImages: true|" /etc/zfsbootmenu/config.yaml'

vboxmanage controlvm DebianZFS keyboardputstring 'apt install --yes libconfig-inifiles-perl libsort-versions-perl libboolean-perl libyaml-pp-perl fzf mbuffer kexec-tools dracut-core'

vboxmanage controlvm DebianZFS keyboardputstring 'update-initramfs -c -k all'
vboxmanage controlvm DebianZFS keyboardputstring 'generate-zbm'
vboxmanage controlvm DebianZFS keyboardputstring 'echo "Boot default"  "spl_hostid=$( hostid ) zbm.prefer=zroot zbm.skip loglevel=4" > /boot/efi/EFI/debian/refind_linux.conf && echo "Boot to menu"  "spl_hostid=$( hostid ) zbm.prefer=zroot zbm.show loglevel=4" >> /boot/efi/EFI/debian/refind_linux.conf'

## Could it be that I messed up here? `zpool export -a` didn't work because it was busy...
## Then I shutdown the machine via VirtualBox "Send the shutdown signal"
# exit
# cd
# umount -R /mnt
# zpool export -a
# reboot

[ahesford]

I don't use Debian or VirtualBox, but it seems like the primary issue here is that VirtualBox hasn't been told how to boot rEFInd.

[Jip-Hop]

You're right. I haven't fixed this issue yet but to confirm the installation works I manually run fs1: and cd efi/refind and run refind_x64.efi to boot into refind and then zfsbootmenu. I played with creating snapshots from Debian and then rolling back via zfsbootmenu. Seems to work as advertised! 🙂