diff --git a/docs/Getting Started/Arch Linux/3-live.rst b/docs/Getting Started/Arch Linux/3-live.rst deleted file mode 100644 index 7df9d4583..000000000 --- a/docs/Getting Started/Arch Linux/3-live.rst +++ /dev/null @@ -1,49 +0,0 @@ -.. highlight:: sh - -Live image -============ - -Latest live image might contain a kernel incompatible with -ZFS. Check the compatibility with the following procedure. - -#. Choose a mirror:: - - https://archlinux.org/mirrorlist/all/ - https://gitea.artixlinux.org/packagesA/artix-mirrorlist/src/branch/master/trunk/mirrorlist - -#. Check the build date of the - latest Arch Linux live image:: - - https://mirrors.dotsrc.org/archlinux/iso/latest/ - https://mirrors.dotsrc.org/artix-linux/iso/ - # archlinux-2021.01.01-x86_64.iso - -#. Check the kernel version of the live image:: - - https://archive.archlinux.org/repos/2021/01/01/core/os/x86_64 - https://archive.artixlinux.org/repos/2021/01/01/system/os/x86_64 - # linux-5.10.3.arch1-1-x86_64.pkg.tar.zst - -#. Check latest zfs-dkms package version:: - - https://archzfs.com/archzfs/x86_64/ - # zfs-dkms-2.0.1-1-x86_64.pkg.tar.zst - # zfs-linux-2.0.1_5.10.10.arch1.1-1-x86_64.pkg.tar.zst - -#. Visit OpenZFS release page:: - - curl -L https://github.com/openzfs/zfs/raw/zfs-2.0.1/META \ - | grep Linux - # Linux-Maximum: 5.10 - # Linux-Minimum: 3.10 - - - If compatible, download the latest live image:: - - https://mirrors.dotsrc.org/archlinux/iso/latest/archlinux-2021.01.01-x86_64.iso - https://mirrors.dotsrc.org/artix-linux/iso/artix-base-openrc-20210101-x86_64.iso - - - If not compatible, use an older live image and verify that it contains - a supported kernel using the above method:: - - https://mirrors.dotsrc.org/archlinux/iso/ - https://iso.artixlinux.org/archived-isos.php diff --git a/docs/Getting Started/Arch Linux/Arch Linux Root on ZFS.rst b/docs/Getting Started/Arch Linux/Arch Linux Root on ZFS.rst index 52f03ff62..61d5c1efd 100644 --- a/docs/Getting Started/Arch Linux/Arch Linux Root on ZFS.rst +++ b/docs/Getting Started/Arch Linux/Arch Linux Root on ZFS.rst @@ -1,6 +1,6 @@ Arch Linux Root on ZFS ====================== -`Start here `__. +`Start here `__. Contents -------- diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/0-overview.rst b/docs/Getting Started/Arch Linux/Root on ZFS/0-overview.rst deleted file mode 100644 index aca8b0b50..000000000 --- a/docs/Getting Started/Arch Linux/Root on ZFS/0-overview.rst +++ /dev/null @@ -1,151 +0,0 @@ -.. highlight:: sh - -Overview -====================== -This document describes how to install Arch Linux with ZFS as root -file system. - -Caution -~~~~~~~ - -- This guide wipes entire physical disks. Back up existing data. -- `GRUB does not and - will not work on 4Kn drive with legacy (BIOS) booting. - `__ - -Partition layout -~~~~~~~~~~~~~~~~ - -GUID partition table (GPT) is used. -EFI system partition will be referred to as **ESP** in this document. - -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space | -+======================+======================+=======================+======================+=====================+=======================+=================+ -| File system | | vfat | ZFS | swap | ZFS | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Size | 1M | 4G, or 1G w/o ISO | 4G | depends on RAM size | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Optional encryption | | *Secure Boot* | luks 1 | plain dm-crypt or | ZFS native encryption | | -| | | | | luks2 | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Partition no. | 5 | 1 | 2 | 4 | 3 | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Mount point | | /boot/efi | /boot | | / | | -| | | /boot/efis/disk-part1 | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ - -Dataset layout -~~~~~~~~~~~~~~ - -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| Dataset | canmount | mountpoint | container | notes | -+===========================+======================+======================+=====================================+===========================================+ -| bpool | off | /boot | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool | off | / | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys | off | none | contains BOOT | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys | off | none | contains ROOT | sys is encryptionroot | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA | off | none | contains placeholder "default" | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default/ | on | /home (inherited) | no | | -| home | | | | user datasets, also called "shared | -| | | | | datasets", "persistent datasets"; also | -| | | | | include /var/lib, /srv, ... | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of | -| | | | | noauto, must use fstab to mount | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ - -Encryption -~~~~~~~~~~ - -- Swap - - Swap is always encrypted. By default, swap is encrypted - with plain dm-crypt with key generated from ``/dev/urandom`` - at every boot. Swap content does not persist between reboots. - - LUKS2-encrypted persistent swap can be - enabled after encrypting both boot pool and root pool, see below. - - With persistent swap, hibernation (suspend-to-disk) can be enabled. - -- Root pool - - ZFS native encryption can be optionally enabled for ``rpool/sys`` - and child datasets. - - User should be aware that, ZFS native encryption does not - encrypt some metadata of the datasets. - ZFS native encryption also does not change master key when ``zfs change-key`` is invoked. - Therefore, you should wipe the disk when password is compromised to protect confidentiality. - See `zfs-load-key.8 `__ - and `zfs-change-key.8 `__ - for more information regarding ZFS native encryption. - - Encryption is enabled at dataset creation and can not be disabled later. - Password can be supplied via SSH. - -- Boot pool - - After encrypting root pool, boot pool can also be encrypted with LUKS1. - This protects initrd from attacks and also protects key material in initrd. - - Password must be interactively entered at boot in GRUB. This disables - password with SSH. - -- Bootloader - - Bootloader can not be encrypted. - - However, with Secure Boot, bootloader - can be verified by motherboard firmware to be untempered, - which should be sufficient for most purposes. - - As enabling Secure Boot is device specific, this is not - covered in detail. - -Booting with disk failure -~~~~~~~~~~~~~~~~~~~~~~~~~ - -This guide is written with disk failure in mind. - -If disks used in Root on ZFS pool failed, but -sufficient redundancy for both root pool and boot pool -still exists, the system will still boot normally. - -Swap partition on the failed disk will fail to mount, -after an 1m30s timeout. - -This feature is useful for use cases such -as an unattended remote server. - -Example: - - - System has disks ``n>1`` - - - Installed with mirrored setup - - - Mirrored setup can tolerate up to ``n-1`` disk failures - - - Disconnect one or more disks, keep at least - one disk connected - - - System still boots, but fails to mount swap and - EFI partition diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst b/docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst index d9b54bb6d..651fa4666 100644 --- a/docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst +++ b/docs/Getting Started/Arch Linux/Root on ZFS/1-preparation.rst @@ -6,88 +6,21 @@ Preparation .. contents:: Table of Contents :local: -#. Download live ISO compatible with ZFS: - - #. Check zfs-dkms version - - Visit `archzfs repo `__, - search for ``zfs-dkms`` in page:: - - zfs-dkms-2.1.2-1-x86_64.pkg.tar.zst - - Ignore unstable variants such as zfs-dkms-git or zfs-dkms-rc. - In this case, version number is 2.1.2, between two ``-``. - - #. Check supported kernel version. Plug the version number into the - following link:: - - https://github.com/openzfs/zfs/blob/zfs-2.1.2/META - - Visit the page, the supported max kernel version is:: - - Linux-Maximum: 5.15 - - #. Visit `Arch Linux releases page - `__ and download a - supported version. Note that minor kernel releases does not - affect ZFS compatibility, i. e., both ``5.15.5`` and ``5.15.15`` - are supported by ZFS with ``Linux-Maximum: 5.15``. - -#. Follow `installation guide on Arch wiki `__ - up to **Update the system clock**. - +#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. +#. Download latest `Arch Linux live image `__ and boot from it. +#. Connect to the Internet. #. Set root password or ``/root/.ssh/authorized_keys``. #. Start SSH server:: - systemctl start sshd + systemctl restart sshd #. Connect from another computer:: ssh root@192.168.1.19 - and, most important, enter a bash shell:: - - bash - - This guide is untested with the default shell ``zsh`` in live environment. - -#. Expand live root filesystem:: - - mount -o remount,size=2G /run/archiso/cowspace/ - -#. `Add archzfs repo <../0-archzfs-repo.html>`__. - -#. `Install zfs-dkms in live environment <../2-zfs-dkms.html#installation>`__. - -#. Load zfs kernel module:: - - modprobe zfs - -#. Kernel variant - - Store the kernel variant in a variable. - Available variants in official repo are: - - - linux - - linux-lts - - linux-zen - - linux-hardened +#. Use bash shell. Other shell not tested:: - :: - - INST_LINVAR='linux' - - ``linux-hardened`` does not support hibernation. - -#. Unique pool suffix. ZFS expects pool names to be - unique, therefore it's recommended to create - pools with a unique suffix:: - - INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) - -#. Identify this installation in ZFS filesystem path:: - - INST_ID=arch + bash #. Target disk @@ -95,8 +28,7 @@ Preparation ls /dev/disk/by-id/* - If using virtio as disk bus, use - ``/dev/disk/by-path/*``. + If using virtio as disk bus, use ``/dev/disk/by-path/*``. Declare disk array:: @@ -106,70 +38,57 @@ Preparation DISK='/dev/disk/by-id/disk1' -#. Choose a primary disk. This disk will be used - for primary EFI partition and hibernation, default to - first disk in the array:: +#. Set partition size: - INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ ) + Set swap size. It's `recommended `__ + to setup a swap partition. If you intend to use hibernation, + the minimum should be no less than RAM size. Skip if swap is not needed:: - If disk path contains colon ``:``, this disk - can not be used for hibernation. ``encrypt`` mkinitcpio - hook treats ``:`` as argument separator without a means to - escape this character. + INST_PARTSIZE_SWAP=8 -#. Set vdev topology, possible values are: + Root pool size, use all remaining disk space if not set:: - - (not set, single disk or striped; no redundancy) - - mirror - - raidz1 - - raidz2 - - raidz3 + INST_PARTSIZE_RPOOL= - :: +#. Check kernel version:: - INST_VDEV= + uname -r + #5.18.7-arch1-1 - This will create a single vdev with the topology of your choice. - It is also possible to manually create a pool with multiple vdevs, such as:: +#. Add ZFS repo:: - zpool create --options \ - poolName \ - mirror sda sdb \ - raidz2 sdc ... \ - raidz3 sde ... \ - spare sdf ... + curl -L https://archzfs.com/archzfs.gpg | pacman-key -a - + pacman-key --lsign-key $(curl -L https://git.io/JsfVS) + curl -L https://git.io/Jsfw2 > /etc/pacman.d/mirrorlist-archzfs - Notice the cost of parity when using RAID-Z. See - `here `__ - and `here `__. + tee -a /etc/pacman.conf <<- 'EOF' - For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy. - This guide will use mirrored bpool for multi-disk setup. + #[archzfs-testing] + #Include = /etc/pacman.d/mirrorlist-archzfs - Refer to `zpoolconcepts `__ - and `zpool-create `__ - man pages for details. + [archzfs] + Include = /etc/pacman.d/mirrorlist-archzfs + EOF -#. Set partition size: +#. Find a ZFS package compatible with the kernel: - Set ESP size. ESP contains Live ISO for recovery, - as described in `optional configurations <4-optional-configuration.html>`__:: + Search kernel version string (e.g. 5.18.7) in both pages: - INST_PARTSIZE_ESP=4 # in GB - #INST_PARTSIZE_ESP=1 # if local recovery is not needed + * https://archzfs.com/archive_archzfs/ + * https://archzfs.com/archzfs/x86_64/ - Set boot pool size. To avoid running out of space while using - boot environments, the minimum is 4GB. Adjust the size if you - intend to use multiple kernel/distros:: + Result: https/.../archive_archzfs/zfs-linux-2.1.5_5.18.7.arch1.1-1-x86_64.pkg.tar.zst - INST_PARTSIZE_BPOOL=4 +#. Find compatible zfs-utils package: - Set swap size. It's `recommended `__ - to setup a swap partition. If you intend to use hibernation, - the minimum should be no less than RAM size. Skip if swap is not needed:: + Search ZFS version string (e.g. 2.1.5) in both pages above. - INST_PARTSIZE_SWAP=8 + Result: https/.../archzfs/x86_64/zfs-utils-2.1.5-2-x86_64.pkg.tar.zst - Root pool size, use all remaining disk space if not set:: +#. Download both then install:: - INST_PARTSIZE_RPOOL= + pacman -U link-to-zfs.zst link-to-utils.zst + +#. Load kernel modules:: + + modprobe zfs diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst b/docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst index 1489079c8..1746d916f 100644 --- a/docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst +++ b/docs/Getting Started/Arch Linux/Root on ZFS/2-system-installation.rst @@ -6,46 +6,29 @@ System Installation .. contents:: Table of Contents :local: -#. Optional: wipe solid-state drives with the generic tool - `blkdiscard `__, - to clean previous partition tables and improve performance. +#. Partition the disks:: - All content will be irrevocably destroyed:: + for i in ${DISK}; do - for i in ${DISK}; do - blkdiscard -f $i & - done - wait + sgdisk --zap-all $i - This is a quick operation and should be completed under one - minute. + sgdisk -n1:1M:+1G -t1:EF00 $i - For other device specific methods, see - `Memory cell clearing `__ + sgdisk -n2:0:+4G -t2:BE00 $i -#. Partition the disks. - See `Overview <0-overview.html>`__ for details:: + test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - for i in ${DISK}; do - sgdisk --zap-all $i - sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i - sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - fi - if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then + if test -z $INST_PARTSIZE_RPOOL; then sgdisk -n3:0:0 -t3:BF00 $i else sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i fi + sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i done #. Create boot pool:: - disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done - if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi - zpool create \ -o compatibility=grub2 \ -o ashift=12 \ @@ -59,12 +42,14 @@ System Installation -O xattr=sa \ -O mountpoint=/boot \ -R /mnt \ - bpool_$INST_UUID \ - $INST_VDEV_BPOOL \ + bpool \ + mirror \ $(for i in ${DISK}; do printf "$i-part2 "; done) + If not using a multi-disk setup, remove ``mirror``. + You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` @@ -91,101 +76,49 @@ System Installation -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ - rpool_$INST_UUID \ - $INST_VDEV \ + rpool \ + mirror \ $(for i in ${DISK}; do printf "$i-part3 "; done) - **Notes:** - - - The use of ``ashift=12`` is recommended here because many drives - today have 4 KiB (or larger) physical sectors, even though they - present 512 B logical sectors. Also, a future replacement drive may - have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4 KiB logical sectors (in which case ``ashift=12`` is required). - - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you - do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` - for ``/var/log``, as `journald requires ACLs - `__ - - Setting ``normalization=formD`` eliminates some corner cases relating - to UTF-8 filename normalization. It also implies ``utf8only=on``, - which means that only UTF-8 filenames are allowed. If you care to - support non-UTF-8 filenames, do not use this option. For a discussion - of why requiring UTF-8 filenames may be a bad idea, see `The problems - with enforced UTF-8 only filenames - `__. - - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you - want to tune it (e.g. ``-o recordsize=1M``), see `these - `__ `various - `__ `blog - `__ - `posts - `__. - - Setting ``relatime=on`` is a middle ground between classic POSIX - ``atime`` behavior (with its significant performance impact) and - ``atime=off`` (which provides the best performance by completely - disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat’s documentation - `__ - for further information. - - Setting ``xattr=sa`` `vastly improves the performance of extended - attributes - `__. - Inside ZFS, extended attributes are used to implement POSIX ACLs. - Extended attributes can also be used by user-space applications. - `They are used by some desktop GUI applications. - `__ - `They can be used by Samba to store Windows ACLs and DOS attributes; - they are required for a Samba Active Directory domain controller. - `__ - Note that ``xattr=sa`` is `Linux-specific - `__. If you move your - ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, - extended attributes will not be readable (though your data will be). If - portability of extended attributes is important to you, omit the - ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole - pool, it is probably fine to use it for ``/var/log``. - - Make sure to include the ``-part3`` portion of the drive path. If you - forget that, you are specifying the whole disk, which ZFS will then - re-partition, and you will lose the bootloader partition(s). - -#. This section implements dataset layout as described in `overview <0-overview.html>`__. + If not using a multi-disk setup, remove ``mirror``. + +#. This section implements dataset layout as described in `overview <1-preparation.html>`__. Create root system container: - Unencrypted:: - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + rpool/archlinux - Encrypted: Pick a strong password. Once compromised, changing password will not keep your data safe. See ``zfs-change-key(8)`` for more info:: - zfs create -o canmount=off -o mountpoint=none -o encryption=on -o keylocation=prompt -o keyformat=passphrase rpool_$INST_UUID/$INST_ID - - Create other system datasets:: - - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA - zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default - zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default - zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default - for i in {usr,var,var/lib}; - do - zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - for i in {home,root,srv,usr/local,var/log,var/spool}; - do - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - chmod 750 /mnt/root + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + -o encryption=on \ + -o keylocation=prompt \ + -o keyformat=passphrase \ + rpool/archlinux + + Create system datasets:: + + zfs create -o canmount=on -o mountpoint=/ rpool/archlinux/root + zfs create -o canmount=on -o mountpoint=/home rpool/archlinux/home + zfs create -o canmount=off -o mountpoint=/var rpool/archlinux/var + zfs create -o canmount=on rpool/archlinux/var/lib + zfs create -o canmount=on rpool/archlinux/var/log + + Create boot dataset:: + + zfs create -o canmount=on -o mountpoint=/boot bpool/archlinux #. Format and mount ESP:: @@ -196,65 +129,15 @@ System Installation done mkdir -p /mnt/boot/efi - mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi - -#. Create separate user dataset at ``/home/User``, dateset name can be - changed later:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User - - If needed, snapshot, rollback and other related permissions can be - delegated to the user later. - -#. Create optional program data datasets to omit data from rollback:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www - # for GNOME - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService - # for Docker - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker - # for NFS - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs - # for LXC - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc - # for LibVirt - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt - ##other application - # zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name - - Add other datasets when needed, such as PostgreSQL. + mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi -#. Install base packages:: +#. Install packages:: pacstrap /mnt base vi mandoc grub efibootmgr mkinitcpio -#. Check compatible kernel version:: + CompatibleVer=$(pacman -Si zfs-linux | grep 'Depends On' | sed "s|.*linux=||" | awk '{ print $1 }') + pacstrap -U /mnt https://archive.archlinux.org/packages/l/linux/linux-${CompatibleVer}-x86_64.pkg.tar.zst - INST_LINVER=$(pacman -Si zfs-${INST_LINVAR} \ - | grep 'Depends On' \ - | sed "s|.*${INST_LINVAR}=||" \ - | awk '{ print $1 }') - -#. Install kernel. Download from archive if kernel is not available:: - - if [ ${INST_LINVER} = \ - $(pacman -Si ${INST_LINVAR} | grep Version | awk '{ print $3 }') ]; then - pacstrap /mnt ${INST_LINVAR} - else - pacstrap -U /mnt \ - https://archive.archlinux.org/packages/l/${INST_LINVAR}/${INST_LINVAR}-${INST_LINVER}-x86_64.pkg.tar.zst - fi - - Ignore ``error: command failed to execute correctly``. - -#. Install archzfs package:: - - pacstrap /mnt zfs-$INST_LINVAR zfs-utils - -#. Install firmware:: + pacstrap /mnt zfs-linux zfs-utils pacstrap /mnt linux-firmware intel-ucode amd-ucode - -#. For other optional packages, - see `ArchWiki `__. diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst b/docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst index 1b90f3601..fb9218239 100644 --- a/docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst +++ b/docs/Getting Started/Arch Linux/Root on ZFS/3-system-configuration.rst @@ -6,121 +6,63 @@ System Configuration .. contents:: Table of Contents :local: -#. Set `mkinitcpio zfs hook scan path - `__:: - - echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=${INST_PRIMARY_DISK%/*}\" >> /mnt/etc/default/grub - #. Generate fstab:: - genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab + mkdir -p /mnt/etc/ for i in ${DISK}; do echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab + umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab done - echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - for i in ${DISK}; do - echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab - echo /dev/mapper/${i##*/}-part4-swap none swap defaults 0 0 >> /mnt/etc/fstab - done - fi - - By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails - to mount. This is unnecessary for mirrored EFI boot partitions. - With the above mount options, systemd will skip mounting them at boot, - only mount them on demand when accessed. + echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \ + noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab #. Configure mkinitcpio:: - mv /mnt/etc/mkinitcpio.conf /mnt/etc/mkinitcpio.conf.original - tee /mnt/etc/mkinitcpio.conf <`__. - -#. Enable DHCP on all ethernet ports:: - - tee /mnt/etc/systemd/network/20-default.network <`__. - - Alternatively, install a network manager such as - ``NetworkManager``. - -#. Enable internet time sync:: - - hwclock --systohc - systemctl enable systemd-timesyncd --root=/mnt -#. Interactively set locale, keymap, timezone, hostname and root password:: +#. Enable internet time synchronisation:: - rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD + hwclock --systohc + systemctl enable systemd-timesyncd --root=/mnt - This can be non-interactive, see man page for details:: +#. Set locale, keymap, timezone, hostname and root password:: rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt --force \ - --locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \ - --keymap=us --timezone="Europe/Berlin" --hostname=myHost \ - --root-password=PASSWORD --root-shell=/bin/bash - - ``systemd-firstboot`` has bugs for setting root password, reset it here:: - - arch-chroot /mnt passwd + systemd-firstboot --root=/mnt --prompt --force #. Generate host id:: zgenhostid -f -o /mnt/etc/hostid -#. Ignore kernel updates:: +#. Enable ZFS services:: - sed -i 's/#IgnorePkg/IgnorePkg/' /mnt/etc/pacman.conf - sed -i "/^IgnorePkg/ s/$/ ${INST_LINVAR} ${INST_LINVAR}-headers zfs-${INST_LINVAR} zfs-utils/" /mnt/etc/pacman.conf + systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt - Kernel will be updated manually. See `here <../1-zfs-linux.html#update-kernel>`__. +#. Add archzfs repo:: -#. Enable ZFS services:: + curl -L https://archzfs.com/archzfs.gpg | pacman-key -a - --gpgdir /mnt/etc/pacman.d/gnupg + pacman-key --lsign-key --gpgdir /mnt/etc/pacman.d/gnupg $(curl -L https://git.io/JsfVS) + curl -L https://git.io/Jsfw2 > /mnt/etc/pacman.d/mirrorlist-archzfs - systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt - systemctl disable zfs-mount --root=/mnt + tee -a /mnt/etc/pacman.conf <<- 'EOF' - At boot, datasets on rpool are mounted with ``/etc/fstab``, - which can control the mounting process more precisely than ``zfs-mount.service``. + #[archzfs-testing] + #Include = /etc/pacman.d/mirrorlist-archzfs -#. Chroot:: + [archzfs] + Include = /etc/pacman.d/mirrorlist-archzfs + EOF - echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK - INST_LINVAR=$INST_LINVAR - INST_UUID=$INST_UUID - INST_ID=$INST_ID - INST_VDEV=$INST_VDEV - DISK=\"$DISK\"" > /mnt/root/chroot - history -w /mnt/home/sys-install-pre-chroot.txt - arch-chroot /mnt bash --login -#. Source variables:: +#. Chroot:: - source /root/chroot + history -w /mnt/home/sys-install-pre-chroot.txt + arch-chroot /mnt /usr/bin/env DISK=$DISK bash -#. Apply locales, change if needed:: +#. Generate locales:: echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen locale-gen - -#. `Add archzfs repo <../0-archzfs-repo.html>`__. diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/4-optional-configuration.rst b/docs/Getting Started/Arch Linux/Root on ZFS/4-optional-configuration.rst deleted file mode 100644 index 28ab046b1..000000000 --- a/docs/Getting Started/Arch Linux/Root on ZFS/4-optional-configuration.rst +++ /dev/null @@ -1,316 +0,0 @@ -.. highlight:: sh - -Optional Configuration -====================== - -.. contents:: Table of Contents - :local: - -Skip to `bootloader <5-bootloader.html>`__ section if -no optional configuration is needed. - -Boot environment manager -~~~~~~~~~~~~~~~~~~~~~~~~ - -A boot environment is a dataset which contains a bootable -instance of an operating system. Within the context of this installation, -boot environments can be created on-the-fly to preserve root file system -states before pacman transactions. - -Install an AUR helper of choice then install ``rozb3-pac`` from AUR -for pacman integration:: - - pacman -S --needed git base-devel sudo - echo 'nobody ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/00_nobody - su - nobody -s /bin/bash - mkdir /tmp/build - export HOME=/tmp/build - cd - git clone https://aur.archlinux.org/paru-bin.git - cd paru-bin - makepkg -si - paru -S rozb3-pac - exit - rm /etc/sudoers.d/00_nobody - -If multi-disk setup is used, enable multi-disk -support inside ``/etc/bieaz.cfg``. - -Supply password with SSH -~~~~~~~~~~~~~~~~~~~~~~~~ - -#. Install mkinitcpio tools:: - - pacman -S mkinitcpio-netconf mkinitcpio-dropbear openssh - -#. Store public keys in ``/etc/dropbear/root_key``:: - - vi /etc/dropbear/root_key - -#. Edit mkinitcpio:: - - tee /etc/mkinitcpio.conf <<- 'EOF' - HOOKS=(base udev autodetect modconf block keyboard netconf dropbear zfsencryptssh zfs filesystems) - EOF - -#. Add ``ip=`` to kernel command line:: - - # example DHCP - echo 'GRUB_CMDLINE_LINUX="ip=::::::dhcp"' >> /etc/default/grub - - Details for ``ip=`` can be found at - `here `__. - -#. Generate host keys:: - - ssh-keygen -Am pem - dropbearconvert openssh dropbear /etc/ssh/ssh_host_ed25519_key /etc/dropbear/dropbear_ed25519_host_key - - `mkinitcpio-dropbear - `__ - lacks support for converting ed25519 host key, - `see this pull request - `__. - -Encrypt boot pool -~~~~~~~~~~~~~~~~~~~ -Note: This will disable password with SSH. The password previously set for -root pool will be replaced by keyfile, embedded in initrd. - -#. LUKS password:: - - LUKS_PWD=secure-passwd - - You will need to enter the same password for - each disk at boot. As root pool key is - protected by this password, the previous warning - about password strength still apply. - - Double-check password here. Complete reinstallation is - needed if entered wrong. - -#. Create encryption keys:: - - mkdir /etc/cryptkey.d/ - chmod 700 /etc/cryptkey.d/ - dd bs=32 count=1 if=/dev/urandom of=/etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs - dd bs=32 count=1 if=/dev/urandom of=/etc/cryptkey.d/bpool_$INST_UUID-key-luks - chmod u=r,go= /etc/cryptkey.d/* - -#. Backup boot pool:: - - zfs snapshot -r bpool_$INST_UUID/$INST_ID@pre-luks - zfs send -Rv bpool_$INST_UUID/$INST_ID@pre-luks > /root/bpool_$INST_UUID-${INST_ID}-pre-luks - -#. Unmount EFI partition:: - - umount /boot/efi - - for i in ${DISK}; do - umount /boot/efis/${i##*/}-part1 - done - -#. Destroy boot pool:: - - zpool destroy bpool_$INST_UUID - -#. Create LUKS containers:: - - for i in ${DISK}; do - cryptsetup luksFormat -q --type luks2 --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2 - echo $LUKS_PWD | cryptsetup luksAddKey --pbkdf pbkdf2 --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2 - cryptsetup open ${i}-part2 ${i##*/}-part2-luks-bpool_$INST_UUID --key-file /etc/cryptkey.d/bpool_$INST_UUID-key-luks - echo ${i##*/}-part2-luks-bpool_$INST_UUID ${i}-part2 /etc/cryptkey.d/bpool_$INST_UUID-key-luks discard >> /etc/crypttab - done - - In GRUB 2.06, only the PBKDF2 key derivation function - is supported, thus PBKDF2 is used - for passphrase key slot. Other slots are not affected. - -#. Embed key file in initrd:: - - echo 'FILES=(/etc/cryptkey.d/* )' >> /etc/mkinitcpio.conf - -#. Recreate boot pool with mappers as vdev:: - - disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done - if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi - - - zpool create \ - -o compatibility=grub2 \ - -o ashift=12 \ - -o autotrim=on \ - -O acltype=posixacl \ - -O canmount=off \ - -O compression=lz4 \ - -O devices=off \ - -O normalization=formD \ - -O relatime=on \ - -O xattr=sa \ - -O mountpoint=/boot \ - bpool_$INST_UUID \ - $INST_VDEV_BPOOL \ - $(for i in ${DISK}; do - printf "/dev/mapper/${i##*/}-part2-luks-bpool_$INST_UUID "; - done) - -#. Restore boot pool backup:: - - zfs recv bpool_${INST_UUID}/${INST_ID} < /root/bpool_$INST_UUID-${INST_ID}-pre-luks - rm /root/bpool_$INST_UUID-${INST_ID}-pre-luks - -#. Mount boot dataset and EFI partitions:: - - mount /boot - mount /boot/efi - - for i in ${DISK}; do - mount /boot/efis/${i##*/}-part1 - done - -#. As keys are stored in initrd, - set secure permissions for ``/boot``:: - - chmod 700 /boot - -#. Change root pool password to key file:: - - zfs change-key -l \ - -o keylocation=file:///etc/cryptkey.d/rpool_$INST_UUID-${INST_ID}-key-zfs \ - -o keyformat=raw \ - rpool_$INST_UUID/$INST_ID - -#. Import encrypted boot pool from ``/dev/mapper``:: - - curl -L https://git.io/Jsfwj > /etc/systemd/system/zfs-import-bpool-mapper.service - systemctl enable zfs-import-bpool-mapper.service - -#. Remove ``zfsencryptssh`` hook. - Encrypted boot pool is incompatible with - password by SSH:: - - sed -i 's|zfsencryptssh||g' /etc/mkinitcpio.conf - - If ``zfsencryptssh`` is not removed, initrd will - stuck at ``fail to load key material`` and fail to boot. - -#. Enable GRUB cryptodisk:: - - echo "GRUB_ENABLE_CRYPTODISK=y" >> /etc/default/grub -#. Let GRUB decrypt all LUKS containers on boot:: - - tee -a /etc/grub.d/09_bpool_luks2-decryption <> /etc/fstab - - # comment out entry in crypttab - sed -i "s|^${INST_PRIMARY_DISK##*/}-part4-swap|#${INST_PRIMARY_DISK##*/}-part4-swap|" /etc/crypttab - - # create key and format partition as LUKS container - dd bs=32 count=1 if=/dev/urandom of=${INST_SWAPKEY}; - chmod u=r,go= /etc/cryptkey.d/* - cryptsetup luksFormat -q --type luks2 --key-file ${INST_SWAPKEY} ${INST_PRIMARY_DISK}-part4 - cryptsetup luksOpen ${INST_PRIMARY_DISK}-part4 ${INST_SWAPMAPPER} --key-file ${INST_SWAPKEY} - - # initialize swap space - mkswap /dev/mapper/${INST_SWAPMAPPER} - -#. Optional: after enabling persistent swap partition, - enable hibernation:: - - # add hook in initrd - sed -i 's| zfs | encrypt resume zfs |' /etc/mkinitcpio.conf - # add kernel cmdline to decrypt swap in initrd - echo "GRUB_CMDLINE_LINUX=\" \ - zfs_import_dir=${INST_PRIMARY_DISK%/*} \ - cryptdevice=PARTUUID=$(blkid -s PARTUUID -o value ${INST_PRIMARY_DISK}-part4):${INST_SWAPMAPPER}:allow-discards \ - cryptkey=rootfs:${INST_SWAPKEY} \ - resume=/dev/mapper/${INST_SWAPMAPPER}\"" \ - >> /etc/default/grub - - Note that hibernation might not work with discrete graphics, virtio graphics or - AMD APU integrated graphics. This is not specific to this guide. - - Computer must resume from a continuous swap space, resume - from multiple swap partitions is not supported. - - ``encrypt`` hook can only decrypt one container at boot. - ``sd-encrypt`` can decrypt multiple devices but is - not compatible with ``zfs`` hook. - - Do not touch anything on disk while the computer is - in hibernation, see `kernel documentation - `__. - -Boot Live ISO with GRUB -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -GRUB `can be configured `__ to boot ISO file directly. - -In this section, we will download Live ISO to ESP and configure GRUB to -boot from it. This enables system recovery and re-installation. - -#. Download Live iso to EFI system partition:: - - mkdir /boot/efi/iso - cd /boot/efi/iso - # select a mirror and disk version - # curl -O https://mirrors.ocf.berkeley.edu/archlinux/iso/2021.11.01/archlinux-2021.11.01-x86_64.iso - curl -O https://archlinux.org/iso/2021.11.01/archlinux-2021.11.01-x86_64.iso.sig - gpg --auto-key-retrieve --verify archlinux-2021.11.01-x86_64.iso.sig - - Additionally you can build your own live image - with `archiso package `__. - - GRUB supports verifying checksum. - See `manual page - `__ - for details. - -#. Add custom GRUB entry for ``/boot/efi/iso/archlinux-*.iso``:: - - curl -L https://git.io/Jsfr3 > /etc/grub.d/43_archiso - chmod +x /etc/grub.d/43_archiso - - You can also boot Live ISO for other distros, see `glim - `__ - configurations. - - ISO is not mirrored to other devices due to its size. - Change ``$ESP_MNT`` to adapt to other ESP. - -#. Generate ``grub.cfg`` in the next step. If a new file - has been added later, regenerate ``grub.cfg``. diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst b/docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst index a0d7bdfb9..76610d5e4 100644 --- a/docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst +++ b/docs/Getting Started/Arch Linux/Root on ZFS/5-bootloader.rst @@ -25,13 +25,6 @@ Workarounds have to be applied. echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh source /etc/profile.d/zpool_vdev_name_path.sh - Note that ``sudo`` will not read ``/etc/profile`` and will - not pass variables in parent shell. Consider setting the following - in ``/etc/sudoers``:: - - pacman -S --noconfirm --needed sudo - echo 'Defaults env_keep += "ZPOOL_VDEV_NAME_PATH"' >> /etc/sudoers - #. Pool name missing See `this bug report `__. @@ -43,60 +36,51 @@ Workarounds have to be applied. sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux + Caution: this fix must be applied after every GRUB update and before generating the menu. + Install GRUB ~~~~~~~~~~~~~~~~~~~~ - -#. Generate initrd:: +#. Create empty cache file and generate initrd:: rm -f /etc/zfs/zpool.cache touch /etc/zfs/zpool.cache chmod a-w /etc/zfs/zpool.cache chattr +i /etc/zfs/zpool.cache - mkinitcpio -P -#. Create GRUB boot directory, in ESP and boot pool:: - - mkdir -p /boot/efi/EFI/arch - mkdir -p /boot/grub - - Boot environment-specific configuration (kernel, etc) - is stored in ``/boot/grub/grub.cfg``, enabling rollback. - -#. When in doubt, install both legacy boot - and EFI. + mkinitcpio -P #. If using legacy booting, install GRUB to every disk:: for i in ${DISK}; do - grub-install --boot-directory /boot/efi/EFI/arch --target=i386-pc $i + grub-install --target=i386-pc $i done #. If using EFI:: - grub-install --boot-directory /boot/efi/EFI/arch --efi-directory /boot/efi/ - grub-install --boot-directory /boot/efi/EFI/arch --efi-directory /boot/efi/ --removable - for i in ${DISK}; do - efibootmgr -cgp 1 -l "\EFI\arch\grubx64.efi" \ - -L "arch-${i##*/}" -d ${i} - done + grub-install --target x86_64-efi + grub-install --target x86_64-efi --removable + for i in ${DISK}; do + efibootmgr -cgp 1 -l "\EFI\arch\grubx64.efi" \ + -L "arch-${i##*/}" -d ${i} + done + +#. Generate GRUB Menu: -#. Generate GRUB Menu:: + Generate menu:: - grub-mkconfig -o /boot/efi/EFI/arch/grub/grub.cfg - cp /boot/efi/EFI/arch/grub/grub.cfg /boot/grub/grub.cfg + echo GRUB_CMDLINE_LINUX=\"zfs_import_dir=/dev/disk/by-id/\" >> /etc/default/grub + grub-mkconfig -o /boot/grub/grub.cfg + cp /boot/grub/grub.cfg /boot/efi/EFI/arch/ #. For both legacy and EFI booting: mirror ESP content:: ESP_MIRROR=$(mktemp -d) + unalias -a cp -r /boot/efi/EFI $ESP_MIRROR for i in /boot/efis/*; do cp -r $ESP_MIRROR/EFI $i done - -Enable Secure Boot ----------------------------- - -This is optional. `See Arch Wiki article `__. + rm -rf $ESP_MIRROR Finish Installation ~~~~~~~~~~~~~~~~~~~~ @@ -105,82 +89,11 @@ Finish Installation exit -#. Take a snapshot of the clean installation for future use:: - - zfs snapshot -r rpool_$INST_UUID/$INST_ID@install - zfs snapshot -r bpool_$INST_UUID/$INST_ID@install - -#. Unmount EFI system partition:: - - umount /mnt/boot/efi - umount /mnt/boot/efis/* - #. Export pools:: - zpool export bpool_$INST_UUID - zpool export rpool_$INST_UUID + umount -Rl /mnt + zpool export -a #. Reboot:: reboot - -Post installaion -~~~~~~~~~~~~~~~~ - -#. If you have other data pools, generate list of datasets for `zfs-mount-generator - `__ to mount them at boot:: - - DATA_POOL='tank0 tank1' - - # tab-separated zfs properties - # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh - export \ - PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\ - ,readonly,setuid,nbmand,encroot,keylocation" - - for i in $DATA_POOL; do - zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i - done - -#. After reboot, consider adding a normal user:: - - # with root permissions - sudo -i - - # store user name in a variable - myUser=UserName - - # rename default `User` to new user name - zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser} - - # update entry in fstab - sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab - - # add user - useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser} - - # delegate snapshot and destroy permissions of the home dataset to - # new user - zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser} - - # fix permissions - chown --recursive ${myUser}:${myUser} /home/${myUser} - chmod 700 /home/${myUser} - - # set new password for user - passwd ${myUser} - - Set up cron job to snapshot user home everyday:: - - pacman -S cronie - systemctl enable --now cronie - crontab -eu ${myUser} - #@daily zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6) - zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2) - - Install package groups:: - - pacman -Sg # query package groups - pacman -S 'gnome' - pacman -S 'plasma' - diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/6-recovery.rst b/docs/Getting Started/Arch Linux/Root on ZFS/6-recovery.rst deleted file mode 100644 index 8d184ac8a..000000000 --- a/docs/Getting Started/Arch Linux/Root on ZFS/6-recovery.rst +++ /dev/null @@ -1,211 +0,0 @@ -.. highlight:: sh - -Recovery -====================== - -.. contents:: Table of Contents - :local: - -GRUB Tips -------------- - -Boot from GRUB rescue -~~~~~~~~~~~~~~~~~~~~~~~ - -If bootloader file is damaged, it's still possible -to boot computer with GRUB rescue image. - -This section is also applicable if you are in -``grub rescue>``. - -#. On another computer, generate rescue image with:: - - pacman -S --needed mtools libisoburn grub - grub-install - grub-mkrescue -o grub-rescue.img - dd if=grub-rescue.img of=/dev/your-usb-stick - - Boot computer from the rescue media. - Both legacy and EFI mode are supported. - - Or `download generated GRUB rescue image `__. - -#. List available disks with ``ls`` command:: - - grub> ls (hd # press tab - Possible devices are: - - hd0 hd1 hd2 hd3 - -#. List partitions by pressing tab key: - - .. code-block:: text - - grub> ls (hd0 # press tab - Possible partitions are: - - Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB - Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB - Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB - - - If boot pool is encrypted: - - Unlock it with ``cryptomount``:: - - grub> insmod luks - grub> cryptomount hd0,gpt2 - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36): - Slot 1 opened - - Unlocked LUKS container is ``(crypto0)``: - - .. code-block:: text - - grub> ls (crypto0) - Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - - - If boot pool is not encrypted: - - .. code-block:: text - - grub> ls (hd0,gpt2) - Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - -#. List boot environments nested inside ``bpool/$INST_ID/BOOT``:: - - grub> ls (crypto0)/sys/BOOT - @/ default/ be0/ - -#. Instruct GRUB to load configuration from ``be0`` boot environment:: - - grub> prefix=(crypto0)/sys/BOOT/be0/@/grub - grub> configfile $prefix/grub.cfg - -#. GRUB menu should now appear. - -#. After entering system, `reinstall GRUB <#grub-installation>`__. - -Switch GRUB prefix when disk fails -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you are using LUKS encrypted boot pool with multiple disks, -the primary disk failed, GRUB will fail to load configuration. - -If there's still enough redundancy for the boot pool, try fix -GRUB with the following method: - -#. Ensure ``Slot 1 opened`` message - is shown - - .. code-block:: text - - Welcome to GRUB! - - error: no such cryptodisk found. - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): - Slot 1 opened - error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found. - Entering rescue mode... - grub rescue> - - If ``error: access denied.`` is shown, - try re-enter password with:: - - grub rescue> cryptomount hd0,gpt2 - -#. Check prefix:: - - grub rescue > set - # prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub - # root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf - -#. Set correct ``prefix`` and ``root`` by replacing - ``cryptouuid/UUID`` with ``crypto0``:: - - grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub - grub rescue> root=crypto0 - -#. Boot GRUB:: - - grub rescue> insmod normal - grub rescue> normal - - GRUB should then boot normally. - -#. After entering system, edit ``/etc/fstab`` to promote - one backup to ``/boot/efi``. - -#. Make the change to ``prefix`` and ``root`` - permanent by `reinstalling GRUB <#grub-installation>`__. - -Access system in chroot ------------------------ - -#. Go through `preparation <1-preparation.html>`__. - -#. Import and unlock root and boot pool:: - - zpool import -NR /mnt rpool_$INST_UUID - zpool import -NR /mnt bpool_$INST_UUID - - If using password:: - - zfs load-key rpool_$INST_UUID/$INST_ID - - If using keyfile:: - - zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID - -#. Find the current boot environment:: - - zfs list - BE=default - -#. Mount root filesystem:: - - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE - -#. chroot into the system:: - - arch-chroot /mnt /bin/bash --login - zfs mount -a - mount -a - -#. Finish rescue. See `finish installation <5-bootloader.html#finish-installation>`__. - -Backup and migrate existing installation ----------------------------------------- -With the help of `zfs send -`__ -it is relatively easy to perform a system backup and migration. - -#. Create a snapshot of root file system:: - - zfs snapshot -r rpool/arch@backup - zfs snapshot -r bpool/arch@backup - -#. Save snapshot to a file or pipe to SSH:: - - zfs send --options rpool/arch@backup > /backup/arch-rpool - zfs send --options bpool/arch@backup > /backup/arch-bpool - -#. Re-create partitions and root/boot - pool on target system. - -#. Restore backup:: - - zfs recv rpool_new/arch < /backup/arch-rpool - zfs recv bpool_new/arch < /backup/arch-bpool - -#. Chroot and reinstall bootloader. - -#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg`` - and ``/etc/zfs/zfs-list.cache/*``. - -#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``. diff --git a/docs/Getting Started/Arch Linux/Root on ZFS/snippets/43_archiso.txt b/docs/Getting Started/Arch Linux/Root on ZFS/snippets/43_archiso.txt deleted file mode 100644 index 98747e03e..000000000 --- a/docs/Getting Started/Arch Linux/Root on ZFS/snippets/43_archiso.txt +++ /dev/null @@ -1,102 +0,0 @@ -#!/bin/sh - -# mountpoint of vfat-formatted partition -ESP_MNT=/boot/efi - -# path to iso files relative to the partition -ISO_REL=/iso - -# absolute path to iso files -ISO_PATH=${ESP_MNT}/${ISO_REL} - -# df command needs warm up due to systemd mount-on-demand -ls $ISO_PATH 1> /dev/null - -# vfat partition UUID -ESP_UUID=$(blkid -s UUID -o value $(df --output=source ${ISO_PATH} | tail -n +2)) - -cat <`__. +`Start here `__. .. toctree:: :maxdepth: 1 diff --git a/docs/Getting Started/Fedora/Fedora Root on ZFS.rst b/docs/Getting Started/Fedora/Fedora Root on ZFS.rst index e4b54b33b..88708777f 100644 --- a/docs/Getting Started/Fedora/Fedora Root on ZFS.rst +++ b/docs/Getting Started/Fedora/Fedora Root on ZFS.rst @@ -1,6 +1,6 @@ Fedora Root on ZFS ====================== -`Start here `__. +`Start here `__. Contents -------- diff --git a/docs/Getting Started/Fedora/Root on ZFS/0-overview.rst b/docs/Getting Started/Fedora/Root on ZFS/0-overview.rst deleted file mode 100644 index dce886376..000000000 --- a/docs/Getting Started/Fedora/Root on ZFS/0-overview.rst +++ /dev/null @@ -1,141 +0,0 @@ -.. highlight:: sh - -Overview -====================== -This document describes how to install Fedora with ZFS as root -file system. - -Caution -~~~~~~~ -- With less than 4GB RAM, DKMS might fail to build - in live environment. -- This guide wipes entire physical disks. Back up existing data. -- `GRUB does not and - will not work on 4Kn drive with legacy (BIOS) booting. - `__ - -Partition layout -~~~~~~~~~~~~~~~~ - -GUID partition table (GPT) is used. -EFI system partition will be referred to as **ESP** in this document. - -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space | -+======================+======================+=======================+======================+=====================+=======================+=================+ -| File system | | vfat | ZFS | swap | ZFS | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Size | 1M | 2G | 4G | depends on RAM size | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | | -| | | | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Partition no. | 5 | 1 | 2 | 4 | 3 | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Mount point | | /boot/efi | /boot | | / | | -| | | /boot/efis/disk-part1 | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ - -Dataset layout -~~~~~~~~~~~~~~ - -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| Dataset | canmount | mountpoint | container | notes | -+===========================+======================+======================+=====================================+===========================================+ -| bpool | off | /boot | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool | off | / | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys | off | none | contains BOOT | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys | off | none | contains ROOT | sys is encryptionroot | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA | off | none | contains placeholder "default" | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default/ | on | /home (inherited) | no | | -| home | | | | user datasets, also called "shared | -| | | | | datasets", "persistent datasets"; also | -| | | | | include /var/lib, /srv, ... | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of | -| | | | | noauto, must use fstab to mount | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ - -Encryption -~~~~~~~~~~ - -- Swap - - Swap is always encrypted. By default, swap is encrypted - with plain dm-crypt with key generated from ``/dev/urandom`` - at every boot. Swap content does not persist between reboots. - -- Root pool - - ZFS native encryption can be optionally enabled for ``rpool/sys`` - and child datasets. - - User should be aware that, ZFS native encryption does not - encrypt some metadata of the datasets. - ZFS native encryption also does not change master key when ``zfs change-key`` is invoked. - Therefore, you should wipe the disk when password is compromised to protect confidentiality. - See `zfs-load-key.8 `__ - and `zfs-change-key.8 `__ - for more information regarding ZFS native encryption. - - Encryption is enabled at dataset creation and can not be disabled later. - -- Boot pool - - Boot pool can not be encrypted. - -- Bootloader - - Bootloader can not be encrypted. - - However, with Secure Boot, bootloader - can be verified by motherboard firmware to be untempered, - which should be sufficient for most purposes. - - Secure Boot is not supported out-of-the-box due to ZFS module. - -Booting with disk failure -~~~~~~~~~~~~~~~~~~~~~~~~~ - -This guide is written with disk failure in mind. - -If disks used in Root on ZFS pool failed, but -sufficient redundancy for both root pool and boot pool -still exists, the system will still boot normally. - -Swap partition on the failed disk will fail to mount, -after an 1m30s timeout. - -This feature is useful for use cases such -as an unattended remote server. - -Example: - - - System has disks ``n>1`` - - - Installed with mirrored setup - - - Mirrored setup can tolerate up to ``n-1`` disk failures - - - Disconnect one or more disks, keep at least - one disk connected - - - System still boots, but fails to mount swap and - EFI partition diff --git a/docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst b/docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst index 6ca0d736f..953c02a1c 100644 --- a/docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst +++ b/docs/Getting Started/Fedora/Root on ZFS/1-preparation.rst @@ -7,75 +7,26 @@ Preparation :local: #. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. -#. Download a variant of Fedora 36 live image - and boot from it. - - - `Fedora Workstation (GNOME) `__ - - `Fedora Spins (Xfce, i3, ...) `__ - +#. Download a variant of `Fedora live image + `__ and boot from it. +#. Connect to the Internet. #. Set root password or ``/root/.ssh/authorized_keys``. #. Start SSH server:: echo PermitRootLogin yes >> /etc/ssh/sshd_config - systemctl start sshd + systemctl restart sshd #. Connect from another computer:: ssh root@192.168.1.19 -#. Temporarily set SELinux to permissive in live environment:: - - setenforce 0 - - SELinux will be enabled on the installed system. - -#. Install ``kernel-devel``:: - - source /etc/os-release - dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/${VERSION_ID}/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm - -#. Add ZFS repo:: - - dnf install -y https://zfsonlinux.org/fedora/zfs-release.fc${VERSION_ID}.noarch.rpm - -#. If zfs-fuse from official Fedora repo is installed, remove it first. It is not maintained and should not be used under any circumstance:: - - rpm -e --nodeps zfs-fuse - -#. Install ZFS packages:: - - dnf install -y zfs - -#. Load kernel modules:: - - modprobe zfs - -#. Install helper script and partition tool:: - - dnf install -y arch-install-scripts gdisk dosfstools - -#. Target Fedora version:: - - INST_FEDORA_VER=${VERSION_ID} - -#. Unique pool suffix. ZFS expects pool names to be - unique, therefore it's recommended to create - pools with a unique suffix:: - - INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) - -#. Identify this installation in ZFS filesystem path:: - - INST_ID=fedora - #. Target disk List available disks with:: ls /dev/disk/by-id/* - If using virtio as disk bus, use - ``/dev/disk/by-path/*``. + If using virtio as disk bus, use ``/dev/disk/by-path/*``. Declare disk array:: @@ -85,63 +36,42 @@ Preparation DISK='/dev/disk/by-id/disk1' -#. Choose a primary disk. This disk will be used - for primary EFI partition and hibernation, default to - first disk in the array:: - - INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ ) - -#. Set vdev topology, possible values are: +#. Set partition size: - - (not set, single disk or striped; no redundancy) - - mirror - - raidz1 - - raidz2 - - raidz3 + Set swap size. It's `recommended `__ + to setup a swap partition. If you intend to use hibernation, + the minimum should be no less than RAM size. Skip if swap is not needed:: - :: + INST_PARTSIZE_SWAP=8 - INST_VDEV= + Root pool size, use all remaining disk space if not set:: - This will create a single vdev with the topology of your choice. - It is also possible to manually create a pool with multiple vdevs, such as:: + INST_PARTSIZE_RPOOL= - zpool create --options \ - poolName \ - mirror sda sdb \ - raidz2 sdc ... \ - raidz3 sde ... \ - spare sdf ... +#. Temporarily set SELinux to permissive in live environment:: - Notice the cost of parity when using RAID-Z. See - `here `__ - and `here `__. + setenforce 0 - For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy. - This guide will use mirrored bpool for multi-disk setup. + SELinux will be enabled on the installed system. - Refer to `zpoolconcepts `__ - and `zpool-create `__ - man pages for details. +#. Add ZFS repo:: -#. Set partition size: + dnf install -y https://zfsonlinux.org/fedora/zfs-release-fedora-2-1.noarch.rpm - Set ESP size:: +#. Check available repos:: - INST_PARTSIZE_ESP=2 # in GB + dnf repolist --all - Set boot pool size. To avoid running out of space while using - boot environments, the minimum is 4GB. Adjust the size if you - intend to use multiple kernel/distros:: +#. Install ZFS packages:: - INST_PARTSIZE_BPOOL=4 + rpm -e --nodeps zfs-fuse + dnf install -y https://dl.fedoraproject.org/pub/fedora/linux/releases/$(source /etc/os-release; echo $VERSION_ID)/Everything/x86_64/os/Packages/k/kernel-devel-$(uname -r).rpm + dnf install -y zfs - Set swap size. It's `recommended `__ - to setup a swap partition. If you intend to use hibernation, - the minimum should be no less than RAM size. Skip if swap is not needed:: +#. Load kernel modules:: - INST_PARTSIZE_SWAP=8 + modprobe zfs - Root pool size, use all remaining disk space if not set:: +#. Install partition tool:: - INST_PARTSIZE_RPOOL= + dnf install -y gdisk dosfstools diff --git a/docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst b/docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst index 208b50512..3d44680b2 100644 --- a/docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst +++ b/docs/Getting Started/Fedora/Root on ZFS/2-system-installation.rst @@ -6,48 +6,29 @@ System Installation .. contents:: Table of Contents :local: -#. Optional: wipe solid-state drives with the generic tool - `blkdiscard `__, - to clean previous partition tables and improve performance. +#. Partition the disks:: - All content will be irrevocably destroyed:: + for i in ${DISK}; do - for i in ${DISK}; do - blkdiscard -f $i & - done - wait + sgdisk --zap-all $i - This is a quick operation and should be completed under one - minute. + sgdisk -n1:1M:+1G -t1:EF00 $i - For other device specific methods, see - `Memory cell clearing `__ + sgdisk -n2:0:+4G -t2:BE00 $i -#. Partition the disks. - See `Overview <0-overview.html>`__ for details:: + test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - for i in ${DISK}; do - sgdisk --zap-all $i - sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i - sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - fi - if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then + if test -z $INST_PARTSIZE_RPOOL; then sgdisk -n3:0:0 -t3:BF00 $i else sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i fi + sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i done #. Create boot pool:: - - disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done - if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi - - zpool create \ -o compatibility=grub2 \ -o ashift=12 \ @@ -61,12 +42,14 @@ System Installation -O xattr=sa \ -O mountpoint=/boot \ -R /mnt \ - bpool_$INST_UUID \ - $INST_VDEV_BPOOL \ + bpool \ + mirror \ $(for i in ${DISK}; do printf "$i-part2 "; done) + If not using a multi-disk setup, remove ``mirror``. + You should not need to customize any of the options for the boot pool. GRUB does not support all of the zpool features. See ``spa_feature_names`` @@ -93,67 +76,15 @@ System Installation -O relatime=on \ -O xattr=sa \ -O mountpoint=/ \ - rpool_$INST_UUID \ - $INST_VDEV \ + rpool \ + mirror \ $(for i in ${DISK}; do printf "$i-part3 "; done) - **Notes:** - - - The use of ``ashift=12`` is recommended here because many drives - today have 4 KiB (or larger) physical sectors, even though they - present 512 B logical sectors. Also, a future replacement drive may - have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4 KiB logical sectors (in which case ``ashift=12`` is required). - - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you - do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` - for ``/var/log``, as `journald requires ACLs - `__ - - Setting ``normalization=formD`` eliminates some corner cases relating - to UTF-8 filename normalization. It also implies ``utf8only=on``, - which means that only UTF-8 filenames are allowed. If you care to - support non-UTF-8 filenames, do not use this option. For a discussion - of why requiring UTF-8 filenames may be a bad idea, see `The problems - with enforced UTF-8 only filenames - `__. - - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you - want to tune it (e.g. ``-o recordsize=1M``), see `these - `__ `various - `__ `blog - `__ - `posts - `__. - - Setting ``relatime=on`` is a middle ground between classic POSIX - ``atime`` behavior (with its significant performance impact) and - ``atime=off`` (which provides the best performance by completely - disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat’s documentation - `__ - for further information. - - Setting ``xattr=sa`` `vastly improves the performance of extended - attributes - `__. - Inside ZFS, extended attributes are used to implement POSIX ACLs. - Extended attributes can also be used by user-space applications. - `They are used by some desktop GUI applications. - `__ - `They can be used by Samba to store Windows ACLs and DOS attributes; - they are required for a Samba Active Directory domain controller. - `__ - Note that ``xattr=sa`` is `Linux-specific - `__. If you move your - ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, - extended attributes will not be readable (though your data will be). If - portability of extended attributes is important to you, omit the - ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole - pool, it is probably fine to use it for ``/var/log``. - - Make sure to include the ``-part3`` portion of the drive path. If you - forget that, you are specifying the whole disk, which ZFS will then - re-partition, and you will lose the bootloader partition(s). - -#. This section implements dataset layout as described in `overview <0-overview.html>`__. + If not using a multi-disk setup, remove ``mirror``. + +#. This section implements dataset layout as described in `overview <1-preparation.html>`__. Create root system container: @@ -162,7 +93,7 @@ System Installation zfs create \ -o canmount=off \ -o mountpoint=none \ - rpool_$INST_UUID/$INST_ID + rpool/redhat - Encrypted: @@ -175,28 +106,19 @@ System Installation -o encryption=on \ -o keylocation=prompt \ -o keyformat=passphrase \ - rpool_$INST_UUID/$INST_ID - - Create other system datasets:: - - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA - zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default - zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default - zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default - for i in {usr,var,var/lib}; - do - zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - for i in {home,root,srv,usr/local,var/log,var/spool}; - do - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - chmod 750 /mnt/root + rpool/redhat + + Create system datasets:: + + zfs create -o canmount=on -o mountpoint=/ rpool/redhat/root + zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home + zfs create -o canmount=off -o mountpoint=/var rpool/redhat/var + zfs create -o canmount=on rpool/redhat/var/lib + zfs create -o canmount=on rpool/redhat/var/log + + Create boot dataset:: + + zfs create -o canmount=on -o mountpoint=/boot bpool/redhat #. Format and mount ESP:: @@ -207,50 +129,15 @@ System Installation done mkdir -p /mnt/boot/efi - mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi - -#. Create separate user dataset at ``/home/User``, dateset name can be - changed later:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User - - If needed, snapshot, rollback and other related permissions can be - delegated to the user later. - -#. Create optional program data datasets to omit data from rollback:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www - # for GNOME - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService - # for Docker - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker - # for NFS - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs - # for LXC - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc - # for LibVirt - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt - ##other application - # zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name - - Add other datasets when needed, such as PostgreSQL. - -#. Install base packages:: - - dnf --installroot=/mnt --releasever=${INST_FEDORA_VER} -y install \ - https://zfsonlinux.org/fedora/zfs-release.fc${INST_FEDORA_VER}.noarch.rpm \ - @core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr cryptsetup \ - kernel kernel-devel python3-dnf-plugin-post-transaction-actions - -#. Install ZFS:: + mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi - dnf --installroot=/mnt -y install zfs zfs-dracut +#. Install packages:: -#. Optional: enable boot environment support and dnf integration:: + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \ + @core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr kernel \ + kernel-devel - dnf --installroot=/mnt copr enable -y m0p/bieaz - dnf --installroot=/mnt install -y bieaz python3-dnf-plugin-rozb3 + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \ + https://zfsonlinux.org/fedora/zfs-release-fedora-2-1.noarch.rpm - If multi-disk setup is used, enable multi-disk - support inside ``/mnt/etc/bieaz.cfg``. + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install zfs zfs-dracut diff --git a/docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst b/docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst index 5b49a6924..ebbb4d14e 100644 --- a/docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst +++ b/docs/Getting Started/Fedora/Root on ZFS/3-system-configuration.rst @@ -8,24 +8,13 @@ System Configuration #. Generate fstab:: - genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab + mkdir -p /mnt/etc/ for i in ${DISK}; do echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab + umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab done - echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - for i in ${DISK}; do - echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab - echo /dev/mapper/${i##*/}-part4-swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0 >> /mnt/etc/fstab - done - fi - - By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails - to mount. This is unnecessary for mirrored EFI boot partitions. - With the above mount options, systemd will skip mounting them at boot, - only mount them on demand when accessed. + echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \ + noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab #. Configure dracut:: @@ -37,25 +26,10 @@ System Configuration echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf fi -#. Enable timezone sync:: - - hwclock --systohc - systemctl enable systemd-timesyncd --root=/mnt - -#. Interactively set locale, keymap, timezone, hostname and root password:: - - rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt --force --prompt --root-password=PASSWORD - - This can be non-interactive, see man page for details:: +#. Set locale, keymap, timezone, hostname and root password:: rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt --force \ - --locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \ - --keymap=us --timezone="Europe/Berlin" --hostname=myHost \ - --root-password=PASSWORD --root-shell=/bin/bash - - ``systemd-firstboot`` have bugs, root password is set below. + systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force #. Generate host id:: @@ -65,16 +39,9 @@ System Configuration dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en - Program will show errors if not installed. - #. Enable ZFS services:: - systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt - systemctl disable zfs-mount --root=/mnt - - At boot, datasets on rpool are mounted with ``/etc/fstab``, - which can control the mounting process more precisely than ``zfs-mount.service``. - + systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt #. By default SSH server is enabled, allowing root login by password, disable SSH server:: @@ -84,22 +51,13 @@ System Configuration #. Chroot:: - echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK - INST_LINVAR=$INST_LINVAR - INST_UUID=$INST_UUID - INST_ID=$INST_ID - unalias -a - INST_VDEV=$INST_VDEV - DISK=\"$DISK\"" > /mnt/root/chroot - #backup bash inputs up to this point - history -w /mnt/home/sys-install-pre-chroot.txt - arch-chroot /mnt bash --login - -#. Source variables:: + m='/dev /proc /sys' + for i in $m; do mount --rbind $i /mnt/$i; done - source /root/chroot + history -w /mnt/home/sys-install-pre-chroot.txt + chroot /mnt /usr/bin/env DISK=$DISK bash --login -#. For SELinux, relabel filesystem on next boot:: +#. For SELinux, relabel filesystem on reboot:: fixfiles -F onboot diff --git a/docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst b/docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst index 1f8764423..25088105c 100644 --- a/docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst +++ b/docs/Getting Started/Fedora/Root on ZFS/5-bootloader.rst @@ -36,6 +36,8 @@ Workarounds have to be applied. sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux + Caution: this fix must be applied after every GRUB update and before generating the menu. + Install GRUB ~~~~~~~~~~~~~~~~~~~~ @@ -43,37 +45,26 @@ Install GRUB echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf -#. Generate initrd:: +#. Create empty cache file and generate initrd:: rm -f /etc/zfs/zpool.cache touch /etc/zfs/zpool.cache chmod a-w /etc/zfs/zpool.cache chattr +i /etc/zfs/zpool.cache + for directory in /lib/modules/*; do kernel_version=$(basename $directory) dracut --force --kver $kernel_version done -#. Disable BLS:: - - echo "GRUB_ENABLE_BLSCFG=false" >> /etc/default/grub - -#. Create GRUB boot directory, in ESP and boot pool:: - - mkdir -p /boot/efi/EFI/fedora # EFI GRUB dir - mkdir -p /boot/efi/EFI/fedora/grub2 # legacy GRUB dir - mkdir -p /boot/grub2 +#. Load ZFS modules and disable BLS:: - Boot environment-specific configuration (kernel, etc) - is stored in ``/boot/grub2/grub.cfg``, enabling rollback. - -#. When in doubt, install both legacy boot - and EFI. + echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub #. If using legacy booting, install GRUB to every disk:: for i in ${DISK}; do - grub2-install --boot-directory /boot/efi/EFI/fedora --target=i386-pc $i + grub2-install --target=i386-pc $i done #. If using EFI:: @@ -82,13 +73,14 @@ Install GRUB efibootmgr -cgp 1 -l "\EFI\fedora\shimx64.efi" \ -L "fedora-${i##*/}" -d ${i} done - cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora + cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/fedora/ + +#. Generate GRUB Menu: -#. Generate GRUB Menu:: + Generate menu:: - grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg - cp /boot/efi/EFI/fedora/grub.cfg /boot/efi/EFI/fedora/grub2/grub.cfg - cp /boot/efi/EFI/fedora/grub.cfg /boot/grub2/grub.cfg + grub2-mkconfig -o /boot/grub2/grub.cfg + cp /boot/grub2/grub.cfg /boot/efi/EFI/fedora/ #. For both legacy and EFI booting: mirror ESP content:: @@ -98,55 +90,14 @@ Install GRUB for i in /boot/efis/*; do cp -r $ESP_MIRROR/EFI $i done + rm -rf $ESP_MIRROR -#. Automatically regenerate GRUB menu on kernel update:: - - tee /etc/dnf/plugins/post-transaction-actions.d/00-update-grub-menu-for-kernel.action </dev/null - # kernel-core package contains vmlinuz and initramfs - # change package name if non-standard kernel is used - kernel-core:in:/usr/local/sbin/update-grub-menu.sh - kernel-core:out:/usr/local/sbin/update-grub-menu.sh - EOF - - tee /usr/local/sbin/update-grub-menu.sh <<-'EOF' >/dev/null - #!/bin/sh - export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - export ZPOOL_VDEV_NAME_PATH=YES - source /etc/os-release - grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg - cp /boot/efi/EFI/${ID}/grub.cfg /boot/efi/EFI/${ID}/grub2/grub.cfg - cp /boot/efi/EFI/${ID}/grub.cfg /boot/grub2/grub.cfg - ESP_MIRROR=$(mktemp -d) - cp -r /boot/efi/EFI $ESP_MIRROR - for i in /boot/efis/*; do - cp -r $ESP_MIRROR/EFI $i - done - rm -rf $ESP_MIRROR - EOF - - chmod +x /usr/local/sbin/update-grub-menu.sh - -#. Notes for GRUB on Fedora - - To support Secure Boot, GRUB has been heavily modified by Fedora, - namely: - - - ``grub2-install`` is `disabled for UEFI `__ - - Only a static, signed version of bootloader is copied to EFI system partition - - This signed bootloader does not have built-in support for either ZFS or LUKS containers - - This signed bootloader only loads configuration from ``/boot/efi/EFI/fedora/grub.cfg`` - - Unrelated to Secure Boot, GRUB has also been modified to provide optional - support for `systemd bootloader specification (bls) `__. - Currently ``blscfg.mod`` is incompatible with root on ZFS. +#. Notes for GRUB on RHEL As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade. Or else the new kernel will not be recognized and system will boot the old kernel on reboot. - Also see `Fedora docs for GRUB - `__. - Finish Installation ~~~~~~~~~~~~~~~~~~~~ @@ -154,85 +105,25 @@ Finish Installation exit -#. Take a snapshot of the clean installation for future use:: - - zfs snapshot -r rpool_$INST_UUID/$INST_ID@install - zfs snapshot -r bpool_$INST_UUID/$INST_ID@install - -#. Unmount EFI system partition:: - - umount /mnt/boot/efi - umount /mnt/boot/efis/* - #. Export pools:: - zpool export bpool_$INST_UUID - zpool export rpool_$INST_UUID + umount -Rl /mnt + zpool export -a #. Reboot:: reboot -Post installaion -~~~~~~~~~~~~~~~~ - -#. If you have other data pools, generate list of datasets for `zfs-mount-generator - `__ to mount them at boot:: +#. On first reboot, the boot process will fail, with failure messages such + as "You are in Emergency Mode...Press Ctrl-D to continue". - DATA_POOL='tank0 tank1' + Wait for the computer to automatically reboot and the problem will be resolved. - # tab-separated zfs properties - # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh - export \ - PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\ - ,readonly,setuid,nbmand,encroot,keylocation" - - for i in $DATA_POOL; do - zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i - done - -#. After reboot, consider adding a normal user:: - - # with root permissions - sudo -i - - # store user name in a variable - myUser=UserName - - # rename default `User` to new user name - zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser} - - # update entry in fstab - sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab - - # add user - useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser} - - # delegate snapshot and destroy permissions of the home dataset to - # new user - zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser} - - # fix permissions - chown --recursive ${myUser}:${myUser} /home/${myUser} - chmod 700 /home/${myUser} - - # fix selinux context - restorecon /home/${myUser} - - # set new password for user - passwd ${myUser} - - Set up cron job to snapshot user home everyday:: - - dnf install cronie - systemctl enable --now crond - crontab -eu ${myUser} - #@daily /usr/sbin/zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6) - zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2) - - Install package groups:: +Post installaion +~~~~~~~~~~~~~~~~ +#. Install package groups:: dnf group list --hidden -v # query package groups - dnf group install 'i3 Desktop' - dnf group install 'Fedora Workstation' # GNOME - dnf group install 'Web Server' + dnf group install @gnome-desktop + +#. Add new user, configure swap. diff --git a/docs/Getting Started/Fedora/Root on ZFS/6-recovery.rst b/docs/Getting Started/Fedora/Root on ZFS/6-recovery.rst deleted file mode 100644 index 1dad98b27..000000000 --- a/docs/Getting Started/Fedora/Root on ZFS/6-recovery.rst +++ /dev/null @@ -1,211 +0,0 @@ -.. highlight:: sh - -Recovery -====================== - -.. contents:: Table of Contents - :local: - -GRUB Tips -------------- - -Boot from GRUB rescue -~~~~~~~~~~~~~~~~~~~~~~~ - -If bootloader file is damaged, it's still possible -to boot computer with GRUB rescue image. - -This section is also applicable if you are in -``grub rescue>``. - -#. On another computer, generate rescue image with:: - - pacman -S --needed mtools libisoburn grub - grub-install - grub-mkrescue -o grub-rescue.img - dd if=grub-rescue.img of=/dev/your-usb-stick - - Boot computer from the rescue media. - Both legacy and EFI mode are supported. - - Or `download generated GRUB rescue image `__. - -#. List available disks with ``ls`` command:: - - grub> ls (hd # press tab - Possible devices are: - - hd0 hd1 hd2 hd3 - -#. List partitions by pressing tab key: - - .. code-block:: text - - grub> ls (hd0 # press tab - Possible partitions are: - - Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB - Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB - Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB - - - If boot pool is encrypted: - - Unlock it with ``cryptomount``:: - - grub> insmod luks - grub> cryptomount hd0,gpt2 - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36): - Slot 1 opened - - Unlocked LUKS container is ``(crypto0)``: - - .. code-block:: text - - grub> ls (crypto0) - Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - - - If boot pool is not encrypted: - - .. code-block:: text - - grub> ls (hd0,gpt2) - Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - -#. List boot environments nested inside ``bpool/$INST_ID/BOOT``:: - - grub> ls (crypto0)/sys/BOOT - @/ default/ be0/ - -#. Instruct GRUB to load configuration from ``be0`` boot environment:: - - grub> prefix=(crypto0)/sys/BOOT/be0/@/grub - grub> configfile $prefix/grub.cfg - -#. GRUB menu should now appear. - -#. After entering system, `reinstall GRUB <#grub-installation>`__. - -Switch GRUB prefix when disk fails -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you are using LUKS encrypted boot pool with multiple disks, -the primary disk failed, GRUB will fail to load configuration. - -If there's still enough redundancy for the boot pool, try fix -GRUB with the following method: - -#. Ensure ``Slot 1 opened`` message - is shown - - .. code-block:: text - - Welcome to GRUB! - - error: no such cryptodisk found. - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): - Slot 1 opened - error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found. - Entering rescue mode... - grub rescue> - - If ``error: access denied.`` is shown, - try re-enter password with:: - - grub rescue> cryptomount hd0,gpt2 - -#. Check prefix:: - - grub rescue > set - # prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub - # root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf - -#. Set correct ``prefix`` and ``root`` by replacing - ``cryptouuid/UUID`` with ``crypto0``:: - - grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub - grub rescue> root=crypto0 - -#. Boot GRUB:: - - grub rescue> insmod normal - grub rescue> normal - - GRUB should then boot normally. - -#. After entering system, edit ``/etc/fstab`` to promote - one backup to ``/boot/efi``. - -#. Make the change to ``prefix`` and ``root`` - permanent by `reinstalling GRUB <#grub-installation>`__. - -Access system in chroot ------------------------ - -#. Go through `preparation <1-preparation.html>`__. - -#. Import and unlock root and boot pool:: - - zpool import -NR /mnt rpool_$INST_UUID - zpool import -NR /mnt bpool_$INST_UUID - - If using password:: - - zfs load-key rpool_$INST_UUID/$INST_ID - - If using keyfile:: - - zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID - -#. Find the current boot environment:: - - zfs list - BE=default - -#. Mount root filesystem:: - - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE - -#. chroot into the system:: - - arch-chroot /mnt /bin/bash --login - zfs mount -a - mount -a - -#. Finish rescue. See `finish installation <#finish-installation>`__. - -Backup and migrate existing installation ----------------------------------------- -With the help of `zfs send -`__ -it is relatively easy to perform a system backup and migration. - -#. Create a snapshot of root file system:: - - zfs snapshot -r rpool/$INST_ID@backup - zfs snapshot -r bpool/$INST_ID@backup - -#. Save snapshot to a file or pipe to SSH:: - - zfs send --options rpool/$INST_ID@backup > /backup/$INST_ID-rpool - zfs send --options bpool/$INST_ID@backup > /backup/$INST_ID-bpool - -#. Re-create partitions and root/boot - pool on target system. - -#. Restore backup:: - - zfs recv rpool_new/$INST_ID < /backup/$INST_ID-rpool - zfs recv bpool_new/$INST_ID < /backup/$INST_ID-bpool - -#. Chroot and reinstall bootloader. - -#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg`` - and ``/etc/zfs/zfs-list.cache/*``. - -#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``. diff --git a/docs/Getting Started/Fedora/index.rst b/docs/Getting Started/Fedora/index.rst index 48b06d916..b525033fb 100644 --- a/docs/Getting Started/Fedora/index.rst +++ b/docs/Getting Started/Fedora/index.rst @@ -90,7 +90,7 @@ Root on ZFS ZFS can be used as root file system for Fedora. An installation guide is available. -`Start here `__. +`Start here `__. .. toctree:: :maxdepth: 1 diff --git a/docs/Getting Started/NixOS/Root on ZFS.rst b/docs/Getting Started/NixOS/Root on ZFS.rst index 72f0de8e0..6737dded9 100644 --- a/docs/Getting Started/NixOS/Root on ZFS.rst +++ b/docs/Getting Started/NixOS/Root on ZFS.rst @@ -1,6 +1,6 @@ NixOS Root on ZFS ======================================= -`Start here `__. +`Start here `__. Contents -------- diff --git a/docs/Getting Started/NixOS/Root on ZFS/0-overview.rst b/docs/Getting Started/NixOS/Root on ZFS/0-overview.rst deleted file mode 100644 index 75598b7e1..000000000 --- a/docs/Getting Started/NixOS/Root on ZFS/0-overview.rst +++ /dev/null @@ -1,142 +0,0 @@ -.. highlight:: sh - -Overview -====================== -This document describes how to install NixOS with ZFS as root -file system. - -Caution -~~~~~~~ -- This guide wipes entire physical disks. Back up existing data. -- `GRUB does not and - will not work on 4Kn drive with legacy (BIOS) booting. - `__ - -Partition layout -~~~~~~~~~~~~~~~~ - -GUID partition table (GPT) is used. -EFI system partition will be referred to as **ESP** in this document. - -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space | -+======================+======================+=======================+======================+=====================+=======================+=================+ -| File system | | vfat | ZFS | swap | ZFS | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Size | 1M | 2G | 4G | depends on RAM size | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | | -| | | | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Partition no. | 5 | 1 | 2 | 4 | 3 | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Mount point | | | /boot | | / | | -| | | /boot/efis/disk-part1 | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ - -Dataset layout -~~~~~~~~~~~~~~ - -The dataset layout used in this guide follows stardard -mutable file positions (``/var``, ``/etc``, ...), but can -still be modified to `a immutable root `__ -after installation. - -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| Dataset | canmount | mountpoint | container | notes | -+===========================+======================+======================+=====================================+===========================================+ -| bpool | off | /boot | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool | off | / | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys | off | none | contains BOOT | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys | off | none | contains ROOT | sys is encryptionroot | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA | off | none | contains placeholder "default" | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/local | off | / | contains /nix datasets | child datsets inherits mountpoint | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default/ | on | /home (inherited) | no | | -| home | | | | user datasets, also called "shared | -| | | | | datasets", "persistent datasets"; also | -| | | | | include /var/lib, /srv, ... | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of | -| | | | | noauto, must use fstab to mount | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ - -Encryption -~~~~~~~~~~ - -- Swap - - Swap is always encrypted. By default, swap is encrypted - with plain dm-crypt with key generated from ``/dev/urandom`` - at every boot. Swap content does not persist between reboots. - -- Root pool - - ZFS native encryption can be optionally enabled for ``rpool/sys`` - and child datasets. - - User should be aware that, ZFS native encryption does not - encrypt some metadata of the datasets. - ZFS native encryption also does not change master key when ``zfs change-key`` is invoked. - Therefore, you should wipe the disk when password is compromised to protect confidentiality. - See `zfs-load-key.8 `__ - and `zfs-change-key.8 `__ - for more information regarding ZFS native encryption. - - Encryption is enabled at dataset creation and can not be disabled later. - -- Boot pool - - Boot pool can not be encrypted. - -- Bootloader - - Bootloader can not be encrypted. - - However, with Secure Boot, bootloader - can be verified by motherboard firmware to be untempered, - which should be sufficient for most purposes. - - Secure Boot is not supported out-of-the-box due to ZFS module. - -Booting with disk failure -~~~~~~~~~~~~~~~~~~~~~~~~~ - -This guide is written with disk failure in mind. - -If disks used in Root on ZFS pool failed, but -sufficient redundancy for both root pool and boot pool -still exists, the system will still boot normally. - -Swap partition on the failed disk will fail to mount, -after an 1m30s timeout. - -This feature is useful for use cases such -as an unattended remote server. - -Example: - - - System has disks ``n>1`` - - - Installed with mirrored setup - - - Mirrored setup can tolerate up to ``n-1`` disk failures - - - Disconnect one or more disks, keep at least - one disk connected - - - System still boots, but fails to mount swap and - EFI partition diff --git a/docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst b/docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst index f8dbfc117..03707681e 100644 --- a/docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst +++ b/docs/Getting Started/NixOS/Root on ZFS/1-preparation.rst @@ -6,41 +6,26 @@ Preparation .. contents:: Table of Contents :local: -#. Download `Minimal ISO image - `__ and boot from it. +#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. +#. Download `NixOS Live Image + `__ and boot from it. +#. Connect to the Internet. +#. Set root password or ``/root/.ssh/authorized_keys``. +#. Start SSH server:: -#. Connect to network. See `NixOS manual `__. - -#. SSH server is enabled by default. To connect, set root password with:: - - sudo passwd + systemctl restart sshd #. Connect from another computer:: ssh root@192.168.1.19 -#. Unique pool suffix. ZFS expects pool names to be - unique, therefore it's recommended to create - pools with a unique suffix:: - - INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) - -#. Identify this installation in ZFS filesystem path:: - - INST_ID=nixos - -#. Root on ZFS configuration file name:: - - INST_CONFIG_FILE='zfs.nix' - #. Target disk List available disks with:: ls /dev/disk/by-id/* - If using virtio as disk bus, use - ``/dev/disk/by-path/*``. + If using virtio as disk bus, use ``/dev/disk/by-path/*``. Declare disk array:: @@ -50,57 +35,8 @@ Preparation DISK='/dev/disk/by-id/disk1' -#. Choose a primary disk. This disk will be used - for primary EFI partition, default to - first disk in the array:: - - INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ ) - -#. Set vdev topology, possible values are: - - - (not set, single disk or striped; no redundancy) - - mirror - - raidz1 - - raidz2 - - raidz3 - - :: - - INST_VDEV= - - This will create a single vdev with the topology of your choice. - It is also possible to manually create a pool with multiple vdevs, such as:: - - zpool create --options \ - poolName \ - mirror sda sdb \ - raidz2 sdc ... \ - raidz3 sde ... \ - spare sdf ... - - Notice the cost of parity when using RAID-Z. See - `here `__ - and `here `__. - - For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy. - This guide will use mirrored bpool for multi-disk setup. - - Refer to `zpoolconcepts `__ - and `zpool-create `__ - man pages for details. - #. Set partition size: - Set ESP size:: - - INST_PARTSIZE_ESP=2 # in GB - - Set boot pool size. To avoid running out of space while using - boot environments, the minimum is 4GB. Adjust the size if you - intend to use multiple kernel/distros:: - - INST_PARTSIZE_BPOOL=4 - Set swap size. It's `recommended `__ to setup a swap partition. If you intend to use hibernation, the minimum should be no less than RAM size. Skip if swap is not needed:: diff --git a/docs/Getting Started/NixOS/Root on ZFS/2-system-configuration.rst b/docs/Getting Started/NixOS/Root on ZFS/2-system-configuration.rst deleted file mode 100644 index 16bec689c..000000000 --- a/docs/Getting Started/NixOS/Root on ZFS/2-system-configuration.rst +++ /dev/null @@ -1,362 +0,0 @@ -.. highlight:: sh - -System Configuration -====================== - -.. contents:: Table of Contents - :local: - -#. Optional: wipe solid-state drives with the generic tool - `blkdiscard `__, - to clean previous partition tables and improve performance. - - All content will be irrevocably destroyed:: - - for i in ${DISK}; do - blkdiscard -f $i & - done - wait - - This is a quick operation and should be completed under one - minute. - - For other device specific methods, see - `Memory cell clearing `__ - -#. Partition the disks. - See `Overview <0-overview.html>`__ for details:: - - for i in ${DISK}; do - sgdisk --zap-all $i - sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i - sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - fi - if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then - sgdisk -n3:0:0 -t3:BF00 $i - else - sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i - fi - sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i - done - -#. Create boot pool:: - - - disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done - if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi - - - zpool create \ - -o compatibility=grub2 \ - -o ashift=12 \ - -o autotrim=on \ - -O acltype=posixacl \ - -O canmount=off \ - -O compression=lz4 \ - -O devices=off \ - -O normalization=formD \ - -O relatime=on \ - -O xattr=sa \ - -O mountpoint=/boot \ - -R /mnt \ - bpool_$INST_UUID \ - $INST_VDEV_BPOOL \ - $(for i in ${DISK}; do - printf "$i-part2 "; - done) - - You should not need to customize any of the options for the boot pool. - - GRUB does not support all of the zpool features. See ``spa_feature_names`` - in `grub-core/fs/zfs/zfs.c - `__. - This step creates a separate boot pool for ``/boot`` with the features - limited to only those that GRUB supports, allowing the root pool to use - any/all features. - - Features enabled with ``-o compatibility=grub2`` can be seen - `here `__. - -#. Create root pool:: - - zpool create \ - -o ashift=12 \ - -o autotrim=on \ - -R /mnt \ - -O acltype=posixacl \ - -O canmount=off \ - -O compression=zstd \ - -O dnodesize=auto \ - -O normalization=formD \ - -O relatime=on \ - -O xattr=sa \ - -O mountpoint=/ \ - rpool_$INST_UUID \ - $INST_VDEV \ - $(for i in ${DISK}; do - printf "$i-part3 "; - done) - - **Notes:** - - - The use of ``ashift=12`` is recommended here because many drives - today have 4 KiB (or larger) physical sectors, even though they - present 512 B logical sectors. Also, a future replacement drive may - have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4 KiB logical sectors (in which case ``ashift=12`` is required). - - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you - do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` - for ``/var/log``, as `journald requires ACLs - `__ - - Setting ``normalization=formD`` eliminates some corner cases relating - to UTF-8 filename normalization. It also implies ``utf8only=on``, - which means that only UTF-8 filenames are allowed. If you care to - support non-UTF-8 filenames, do not use this option. For a discussion - of why requiring UTF-8 filenames may be a bad idea, see `The problems - with enforced UTF-8 only filenames - `__. - - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you - want to tune it (e.g. ``-o recordsize=1M``), see `these - `__ `various - `__ `blog - `__ - `posts - `__. - - Setting ``relatime=on`` is a middle ground between classic POSIX - ``atime`` behavior (with its significant performance impact) and - ``atime=off`` (which provides the best performance by completely - disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat’s documentation - `__ - for further information. - - Setting ``xattr=sa`` `vastly improves the performance of extended - attributes - `__. - Inside ZFS, extended attributes are used to implement POSIX ACLs. - Extended attributes can also be used by user-space applications. - `They are used by some desktop GUI applications. - `__ - `They can be used by Samba to store Windows ACLs and DOS attributes; - they are required for a Samba Active Directory domain controller. - `__ - Note that ``xattr=sa`` is `Linux-specific - `__. If you move your - ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, - extended attributes will not be readable (though your data will be). If - portability of extended attributes is important to you, omit the - ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole - pool, it is probably fine to use it for ``/var/log``. - - Make sure to include the ``-part3`` portion of the drive path. If you - forget that, you are specifying the whole disk, which ZFS will then - re-partition, and you will lose the bootloader partition(s). - -#. This section implements dataset layout as described in `overview <0-overview.html>`__. - - Create root system container: - - - Unencrypted:: - - zfs create \ - -o canmount=off \ - -o mountpoint=none \ - rpool_$INST_UUID/$INST_ID - - - Encrypted: - - Pick a strong password. Once compromised, changing password will not keep your - data safe. See ``zfs-change-key(8)`` for more info:: - - zfs create \ - -o canmount=off \ - -o mountpoint=none \ - -o encryption=on \ - -o keylocation=prompt \ - -o keyformat=passphrase \ - rpool_$INST_UUID/$INST_ID - - Create other system datasets:: - - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA - zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default - zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default - zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/local - zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default - for i in {usr,var,var/lib}; - do - zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - for i in {home,root,srv,usr/local,var/log,var/spool}; - do - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - chmod 750 /mnt/root - for i in {nix,}; do - zfs create -o canmount=on -o mountpoint=/$i rpool_$INST_UUID/$INST_ID/DATA/local/$i - done - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/state - for i in {/etc/nixos,/etc/cryptkey.d}; do - mkdir -p /mnt/state/$i /mnt/$i - mount -o bind /mnt/state/$i /mnt/$i - done - zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/empty - zfs snapshot rpool_$INST_UUID/$INST_ID/ROOT/empty@start - -#. Format and mount ESP:: - - for i in ${DISK}; do - mkfs.vfat -n EFI ${i}-part1 - mkdir -p /mnt/boot/efis/${i##*/}-part1 - mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1 - done - -#. Create optional user data datasets to omit data from rollback:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www - # for GNOME - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService - # for Docker - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker - # for NFS - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs - # for LXC - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc - # for LibVirt - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt - ##other application - # zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name - - Add other datasets when needed, such as PostgreSQL. - -#. Generate initial NixOS system configuration:: - - nixos-generate-config --root /mnt - - This command will generate two files, ``configuration.nix`` - and ``hardware-configuration-zfs.nix``, which will be the starting point - of configuring the system. - -#. Edit config file to import ZFS options:: - - sed -i "s|./hardware-configuration.nix|./hardware-configuration-zfs.nix ./${INST_CONFIG_FILE}|g" /mnt/etc/nixos/configuration.nix - # backup, prevent being overwritten by nixos-generate-config - mv /mnt/etc/nixos/hardware-configuration.nix /mnt/etc/nixos/hardware-configuration-zfs.nix - -#. ZFS options:: - - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} < /mnt/state/etc/machine-id - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <>/mnt/etc/nixos/${INST_CONFIG_FILE} - done - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <`__. + This step creates a separate boot pool for ``/boot`` with the features + limited to only those that GRUB supports, allowing the root pool to use + any/all features. + + Features enabled with ``-o compatibility=grub2`` can be seen + `here `__. + +#. Create root pool:: + + zpool create \ + -o ashift=12 \ + -o autotrim=on \ + -R /mnt \ + -O acltype=posixacl \ + -O canmount=off \ + -O compression=zstd \ + -O dnodesize=auto \ + -O normalization=formD \ + -O relatime=on \ + -O xattr=sa \ + -O mountpoint=/ \ + rpool \ + mirror \ + $(for i in ${DISK}; do + printf "$i-part3 "; + done) + + If not using a multi-disk setup, remove ``mirror``. + +#. This section implements dataset layout as described in `overview <1-preparation.html>`__. + + Create root system container: + + - Unencrypted:: + + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + rpool/nixos + + - Encrypted: + + Pick a strong password. Once compromised, changing password will not keep your + data safe. See ``zfs-change-key(8)`` for more info:: + + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + -o encryption=on \ + -o keylocation=prompt \ + -o keyformat=passphrase \ + rpool/nixos + + Create system datasets:: + + zfs create -o canmount=on -o mountpoint=/ rpool/nixos/root + zfs create -o canmount=on -o mountpoint=/home rpool/nixos/home + zfs create -o canmount=off -o mountpoint=/var rpool/nixos/var + zfs create -o canmount=on rpool/nixos/var/lib + zfs create -o canmount=on rpool/nixos/var/log + + Create boot dataset:: + + zfs create -o canmount=on -o mountpoint=/boot bpool/nixos + +#. Format and mount ESP:: + + for i in ${DISK}; do + mkfs.vfat -n EFI ${i}-part1 + mkdir -p /mnt/boot/efis/${i##*/}-part1 + mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1 + done + + mkdir -p /mnt/boot/efi + mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi diff --git a/docs/Getting Started/NixOS/Root on ZFS/3-optional-configuration.rst b/docs/Getting Started/NixOS/Root on ZFS/3-optional-configuration.rst deleted file mode 100644 index f44a2db77..000000000 --- a/docs/Getting Started/NixOS/Root on ZFS/3-optional-configuration.rst +++ /dev/null @@ -1,235 +0,0 @@ -.. highlight:: sh - -Optional Configuration -====================== - -.. contents:: Table of Contents - :local: - -Skip to `System Installation <./4-system-installation.html>`__ section if -no optional configuration is needed. - -Mail notification for ZFS status -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For headless applications such as NAS, it is useful to set up mail notification -for hardware changes and monitor for scrub results. - -#. Set up an alias for root account:: - - tee -a /state/etc/aliases <> /root/.profile - ''; - }; - }; - EOF - -Encrypt boot pool -~~~~~~~~~~~~~~~~~~~ -Note: This will disable password with SSH. The password previously set for -root pool will be replaced by keyfile, embedded in initrd. - -#. Add package:: - - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} < /mnt/root/bpool_$INST_UUID-${INST_ID}-pre-luks - -#. Unmount EFI partition:: - - for i in ${DISK}; do - umount /mnt/boot/efis/${i##*/}-part1 - done - -#. Destroy boot pool:: - - zpool destroy bpool_$INST_UUID - -#. Create LUKS containers:: - - for i in ${DISK}; do - cryptsetup luksFormat -q --type luks1 --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2 - echo $LUKS_PWD | cryptsetup luksAddKey --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks $i-part2 - cryptsetup open ${i}-part2 ${i##*/}-part2-luks-bpool_$INST_UUID --key-file /mnt/etc/cryptkey.d/bpool_$INST_UUID-key-luks - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <>/mnt/etc/nixos/zfs.nix + done + + tee -a /mnt/etc/nixos/zfs.nix <`__ -for details. - -For timezone, hostname, networking, keyboard layout, etc, -see ``/mnt/etc/nixos/configuration.nix``. - -Set root password ------------------ - -This optional step is an example -of declaratively configuring the system. - -#. Generate password hash:: - - INST_ROOT_PASSWD=$(mkpasswd -m SHA-512 -s) - -#. Declare `initialHashedPassword - `__ - for root user:: - - tee -a /mnt/etc/nixos/${INST_CONFIG_FILE} <`__. - Complete the installation by executing:: - - nixos-enter --root /mnt -- nixos-rebuild boot - -Finish installation -~~~~~~~~~~~~~~~~~~~~ - -#. Take a snapshot of the clean installation for future use:: - - zfs snapshot -r rpool_$INST_UUID/$INST_ID@install - zfs snapshot -r bpool_$INST_UUID/$INST_ID@install - -#. Unmount EFI system partition:: - - umount /mnt/boot/efis/* - -#. Export pools:: - - zpool export bpool_$INST_UUID - zpool export rpool_$INST_UUID - -#. Reboot:: - - reboot - -Upgrade NixOS -~~~~~~~~~~~~~ - -Routine updates within the same major version -============================================= - -Updates within the same major version, such as from [21.11].001 to -[21.11].100, can be done with one of the following commands:: - - # take immediate effect - nixos-rebuild --upgrade switch - - # update upon reboot - nixos-rebuild --upgrade boot - -Upgrade to a newer major version -================================ - -Upgrading to a newer major version involves switching software -distribution channel. - -#. View existing channels, run as root:: - - nix-channel --list - #nixos https://nixos.org/channels/nixos-21.11 - - #this is the major version released around November 2021 - -#. View available channels:: - - w3m https://hydra.nixos.org/project/nixos - -#. Switch to a newer channel (22.05):: - - nix-channel --add https://nixos.org/channels/nixos-22.05 nixos - -#. In ``/etc/nixos/configuration.nix``:: - - system.stateVersion = "22.05"; - - If using Home Manager, in ``~/.config/nixpkgs/home.nix``:: - - home.stateVersion = "22.05"; - -#. Then follow the procedures for updating witin minor versions. - -Immutable root file system -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This section is optional. - -Often, programs generate mutable files in paths such as -``/etc`` and ``/var/lib``. The generated files can be considered a -part of the system state. - -This generated state is not declaratively managed -by NixOS and can not be reproduced from NixOS configuration. - -To ensure that the system state is fully managed by NixOS and reproducible, -we need to periodically purge the system state and let NixOS -regenerate root file system from scratch. - -Also see: `Erase your darlings: -immutable infrastructure for mutable systems `__. - -Save mutable data to alternative path -------------------------------------- - -Before enabling purging on root dataset, we need to back up -essential mutable data first, such as host SSH key and network connections. -Below are some tips. - -- Some programs support specifying another - location for mutable data, such as - Wireguard:: - - networking.wireguard.interfaces.wg0.privateKeyFile = "/state/etc/wireguard/wg0"; - -- For programs without a configurable data path, - `environment.etc `__ - may be used:: - - environment.etc = { - "ssh/ssh_host_rsa_key".source = "/state/etc/ssh/ssh_host_rsa_key"; - } - -- systemd’s tmpfiles.d rules are also an option:: - - systemd.tmpfiles.rules = [ - "L /var/lib/bluetooth - - - - /state/var/lib/bluetooth" - ]; - -- Bind mount:: - - for i in {/etc/nixos,/etc/cryptkey.d}; do - mkdir -p /state/$i /$i - mount -o bind /state/$i /$i - done - nixos-generate-config --show-hardware-config - -Boot from empty root file system --------------------------------- - -After backing up mutable data, you can try switching to -an empty dataset as root file system. - -#. Check current root file system:: - - ROOT_FS=$(df --output=source /|tail -n1) - # rpool/ROOT/default - -#. Set empty file system as root:: - - sed -i "s,${ROOT_FS},${ROOT_FS%/*}/empty,g" /etc/nixos/hardware-configuration-zfs.nix - -#. Apply changes and reboot:: - - nixos-rebuild boot - reboot - -#. If everything went fine, add the output of the following command to configuration:: - - ROOT_FS=$(df --output=source /|tail -n1) - cat <``. - -#. On another computer, generate rescue image with:: - - pacman -S --needed mtools libisoburn grub - grub-install - grub-mkrescue -o grub-rescue.img - dd if=grub-rescue.img of=/dev/your-usb-stick - - Boot computer from the rescue media. - Both legacy and EFI mode are supported. - - Or `download generated GRUB rescue image `__. - -#. List available disks with ``ls`` command:: - - grub> ls (hd # press tab - Possible devices are: - - hd0 hd1 hd2 hd3 - -#. List partitions by pressing tab key: - - .. code-block:: text - - grub> ls (hd0 # press tab - Possible partitions are: - - Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB - Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB - Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB - - - If boot pool is encrypted: - - Unlock it with ``cryptomount``:: - - grub> insmod luks - grub> cryptomount hd0,gpt2 - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36): - Slot 1 opened - - Unlocked LUKS container is ``(crypto0)``: - - .. code-block:: text - - grub> ls (crypto0) - Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - - - If boot pool is not encrypted: - - .. code-block:: text - - grub> ls (hd0,gpt2) - Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - -#. List boot environments nested inside ``bpool/$INST_ID/BOOT``:: - - grub> ls (crypto0)/sys/BOOT - @/ default/ be0/ - -#. Instruct GRUB to load configuration from ``be0`` boot environment:: - - grub> prefix=(crypto0)/sys/BOOT/be0/@/grub - grub> configfile $prefix/grub.cfg - -#. GRUB menu should now appear. - -#. After entering system, reinstall GRUB. - -Switch GRUB prefix when disk fails -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you are using LUKS encrypted boot pool with multiple disks, -the primary disk failed, GRUB will fail to load configuration. - -If there's still enough redundancy for the boot pool, try fix -GRUB with the following method: - -#. Ensure ``Slot 1 opened`` message - is shown - - .. code-block:: text - - Welcome to GRUB! - - error: no such cryptodisk found. - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): - Slot 1 opened - error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found. - Entering rescue mode... - grub rescue> - - If ``error: access denied.`` is shown, - try re-enter password with:: - - grub rescue> cryptomount hd0,gpt2 - -#. Check prefix:: - - grub rescue > set - # prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub - # root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf - -#. Set correct ``prefix`` and ``root`` by replacing - ``cryptouuid/UUID`` with ``crypto0``:: - - grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub - grub rescue> root=crypto0 - -#. Boot GRUB:: - - grub rescue> insmod normal - grub rescue> normal - - GRUB should then boot normally. - -#. After entering system, edit ``/etc/fstab`` to promote - one backup to ``/boot/efi``. - -#. Make the change to ``prefix`` and ``root`` - permanent by `reinstalling GRUB <#grub-installation>`__. diff --git a/docs/Getting Started/NixOS/index.rst b/docs/Getting Started/NixOS/index.rst index ae214cf31..926a9f14c 100644 --- a/docs/Getting Started/NixOS/index.rst +++ b/docs/Getting Started/NixOS/index.rst @@ -55,7 +55,7 @@ Root on ZFS ZFS can be used as root file system for NixOS. An installation guide is available. -`Start here `__. +`Start here `__. .. toctree:: :maxdepth: 1 diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS.rst deleted file mode 100644 index d02d20ccb..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS.rst +++ /dev/null @@ -1,11 +0,0 @@ -Rocky Linux 8 Root on ZFS -======================================= -`Start here `__. - -Contents --------- -.. toctree:: - :maxdepth: 2 - :glob: - - RHEL 8-based distro Root on ZFS/* diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst deleted file mode 100644 index d9ea5135c..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/0-overview.rst +++ /dev/null @@ -1,141 +0,0 @@ -.. highlight:: sh - -Overview -====================== -This document describes how to install RHEL 8-based distro with ZFS as root -file system. - -Caution -~~~~~~~ -- With less than 4GB RAM, DKMS might fail to build - in live environment. -- This guide wipes entire physical disks. Back up existing data. -- `GRUB does not and - will not work on 4Kn drive with legacy (BIOS) booting. - `__ - -Partition layout -~~~~~~~~~~~~~~~~ - -GUID partition table (GPT) is used. -EFI system partition will be referred to as **ESP** in this document. - -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Name | legacy boot | ESP | Boot pool | swap | root pool | remaining space | -+======================+======================+=======================+======================+=====================+=======================+=================+ -| File system | | vfat | ZFS | swap | ZFS | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Size | 1M | 2G | 4G | depends on RAM size | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Optional encryption | | *Secure Boot* | | plain dm-crypt | ZFS native encryption | | -| | | | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Partition no. | 5 | 1 | 2 | 4 | 3 | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ -| Mount point | | /boot/efi | /boot | | / | | -| | | /boot/efis/disk-part1 | | | | | -+----------------------+----------------------+-----------------------+----------------------+---------------------+-----------------------+-----------------+ - -Dataset layout -~~~~~~~~~~~~~~ - -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| Dataset | canmount | mountpoint | container | notes | -+===========================+======================+======================+=====================================+===========================================+ -| bpool | off | /boot | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool | off | / | contains sys | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys | off | none | contains BOOT | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys | off | none | contains ROOT | sys is encryptionroot | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT | off | none | contains boot environments | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA | off | none | contains placeholder "default" | | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default | off | / | contains user datasets | child datsets inherits mountpoint | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/DATA/default/ | on | /home (inherited) | no | | -| home | | | | user datasets, also called "shared | -| | | | | datasets", "persistent datasets"; also | -| | | | | include /var/lib, /srv, ... | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/default | noauto | /boot | no | noauto is used to switch BE. because of | -| | | | | noauto, must use fstab to mount | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/default | noauto | / | no | mounted by initrd zfs hook | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| bpool/sys/BOOT/be1 | noauto | /boot | no | see bpool/sys/BOOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ -| rpool/sys/ROOT/be1 | noauto | / | no | see rpool/sys/ROOT/default | -+---------------------------+----------------------+----------------------+-------------------------------------+-------------------------------------------+ - -Encryption -~~~~~~~~~~ - -- Swap - - Swap is always encrypted. By default, swap is encrypted - with plain dm-crypt with key generated from ``/dev/urandom`` - at every boot. Swap content does not persist between reboots. - -- Root pool - - ZFS native encryption can be optionally enabled for ``rpool/sys`` - and child datasets. - - User should be aware that, ZFS native encryption does not - encrypt some metadata of the datasets. - ZFS native encryption also does not change master key when ``zfs change-key`` is invoked. - Therefore, you should wipe the disk when password is compromised to protect confidentiality. - See `zfs-load-key.8 `__ - and `zfs-change-key.8 `__ - for more information regarding ZFS native encryption. - - Encryption is enabled at dataset creation and can not be disabled later. - -- Boot pool - - Boot pool can not be encrypted. - -- Bootloader - - Bootloader can not be encrypted. - - However, with Secure Boot, bootloader - can be verified by motherboard firmware to be untempered, - which should be sufficient for most purposes. - - Secure Boot is not supported out-of-the-box due to ZFS module. - -Booting with disk failure -~~~~~~~~~~~~~~~~~~~~~~~~~ - -This guide is written with disk failure in mind. - -If disks used in Root on ZFS pool failed, but -sufficient redundancy for both root pool and boot pool -still exists, the system will still boot normally. - -Swap partition on the failed disk will fail to mount, -after an 1m30s timeout. - -This feature is useful for use cases such -as an unattended remote server. - -Example: - - - System has disks ``n>1`` - - - Installed with mirrored setup - - - Mirrored setup can tolerate up to ``n-1`` disk failures - - - Disconnect one or more disks, keep at least - one disk connected - - - System still boots, but fails to mount swap and - EFI partition diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst deleted file mode 100644 index a5b4f1282..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/1-preparation.rst +++ /dev/null @@ -1,149 +0,0 @@ -.. highlight:: sh - -Preparation -====================== - -.. contents:: Table of Contents - :local: - -#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. -#. Download a variant of `Rocky Linux 8.5 Live - ISO `__ and boot from it. - -#. Set root password or ``/root/.ssh/authorized_keys``. -#. Start SSH server:: - - echo PermitRootLogin yes >> /etc/ssh/sshd_config - systemctl restart sshd - -#. Connect from another computer:: - - ssh root@192.168.1.19 - -#. Temporarily set SELinux to permissive in live environment:: - - setenforce 0 - - SELinux will be enabled on the installed system. - -#. Optional: If mirror speed is slow, you can manually pick a fixed mirror - from `mirrorlist `__ - and apply it:: - - sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/* - sed -i 's|^#baseurl=|baseurl=|g' /etc/yum.repos.d/* - sed -i 's|dl.rockylinux.org/$contentdir|mirrors.sjtug.sjtu.edu.cn/rocky|g' /etc/yum.repos.d/* - -#. Add ZFS repo:: - - source /etc/os-release - RHEL_ZFS_REPO=https://zfsonlinux.org/epel/zfs-release.el${VERSION_ID/./_}.noarch.rpm - dnf install -y $RHEL_ZFS_REPO - -#. Install ZFS packages:: - - dnf install -y epel-release - dnf config-manager --disable zfs - dnf config-manager --enable zfs-kmod - dnf install -y zfs - -#. Load kernel modules:: - - modprobe zfs - -#. Install helper script and partition tool:: - - rpm -ivh --nodeps https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/x86_64/os/Packages/a/arch-install-scripts-24-3.fc36.noarch.rpm - dnf install -y gdisk dosfstools - -#. Set RHEL major version:: - - INST_RHEL_VER=${VERSION_ID%%.*} - -#. Unique pool suffix. ZFS expects pool names to be - unique, therefore it's recommended to create - pools with a unique suffix:: - - INST_UUID=$(dd if=/dev/urandom bs=1 count=100 2>/dev/null | tr -dc 'a-z0-9' | cut -c-6) - -#. Identify this installation in ZFS filesystem path:: - - INST_ID=rhel - -#. Target disk - - List available disks with:: - - ls /dev/disk/by-id/* - - If using virtio as disk bus, use - ``/dev/disk/by-path/*``. - - Declare disk array:: - - DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' - - For single disk installation, use:: - - DISK='/dev/disk/by-id/disk1' - -#. Choose a primary disk. This disk will be used - for primary EFI partition, default to - first disk in the array:: - - INST_PRIMARY_DISK=$(echo $DISK | cut -f1 -d\ ) - -#. Set vdev topology, possible values are: - - - (not set, single disk or striped; no redundancy) - - mirror - - raidz1 - - raidz2 - - raidz3 - - :: - - INST_VDEV= - - This will create a single vdev with the topology of your choice. - It is also possible to manually create a pool with multiple vdevs, such as:: - - zpool create --options \ - poolName \ - mirror sda sdb \ - raidz2 sdc ... \ - raidz3 sde ... \ - spare sdf ... - - Notice the cost of parity when using RAID-Z. See - `here `__ - and `here `__. - - For boot pool, which must be readable by GRUB, mirrored vdev should always be used for maximum redundancy. - This guide will use mirrored bpool for multi-disk setup. - - Refer to `zpoolconcepts `__ - and `zpool-create `__ - man pages for details. - -#. Set partition size: - - Set ESP size:: - - INST_PARTSIZE_ESP=2 # in GB - - Set boot pool size. To avoid running out of space while using - boot environments, the minimum is 4GB. Adjust the size if you - intend to use multiple kernel/distros:: - - INST_PARTSIZE_BPOOL=4 - - Set swap size. It's `recommended `__ - to setup a swap partition. If you intend to use hibernation, - the minimum should be no less than RAM size. Skip if swap is not needed:: - - INST_PARTSIZE_SWAP=8 - - Root pool size, use all remaining disk space if not set:: - - INST_PARTSIZE_RPOOL= diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst deleted file mode 100644 index f48c8192f..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/2-system-installation.rst +++ /dev/null @@ -1,271 +0,0 @@ -.. highlight:: sh - -System Installation -====================== - -.. contents:: Table of Contents - :local: - -#. Optional: wipe solid-state drives with the generic tool - `blkdiscard `__, - to clean previous partition tables and improve performance. - - All content will be irrevocably destroyed:: - - for i in ${DISK}; do - blkdiscard $i & - done - wait - - This is a quick operation and should be completed under one - minute. - - For other device specific methods, see - `Memory cell clearing `__ - -#. Partition the disks. - See `Overview <0-overview.html>`__ for details:: - - for i in ${DISK}; do - sgdisk --zap-all $i - sgdisk -n1:1M:+${INST_PARTSIZE_ESP}G -t1:EF00 $i - sgdisk -n2:0:+${INST_PARTSIZE_BPOOL}G -t2:BE00 $i - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i - fi - if [ "${INST_PARTSIZE_RPOOL}" = "" ]; then - sgdisk -n3:0:0 -t3:BF00 $i - else - sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i - fi - sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i - done - -#. Create boot pool:: - - - disk_num=0; for i in $DISK; do disk_num=$(( $disk_num + 1 )); done - if [ $disk_num -gt 1 ]; then INST_VDEV_BPOOL=mirror; fi - - - zpool create \ - -d -o feature@async_destroy=enabled \ - -o feature@bookmarks=enabled \ - -o feature@embedded_data=enabled \ - -o feature@empty_bpobj=enabled \ - -o feature@enabled_txg=enabled \ - -o feature@extensible_dataset=enabled \ - -o feature@filesystem_limits=enabled \ - -o feature@hole_birth=enabled \ - -o feature@large_blocks=enabled \ - -o feature@lz4_compress=enabled \ - -o feature@spacemap_histogram=enabled \ - -o ashift=12 \ - -o autotrim=on \ - -O acltype=posixacl \ - -O canmount=off \ - -O compression=lz4 \ - -O devices=off \ - -O normalization=formD \ - -O relatime=on \ - -O xattr=sa \ - -O mountpoint=/boot \ - -R /mnt \ - bpool_$INST_UUID \ - $INST_VDEV_BPOOL \ - $(for i in ${DISK}; do - printf "$i-part2 "; - done) - - You should not need to customize any of the options for the boot pool. - - GRUB does not support all of the zpool features. See ``spa_feature_names`` - in `grub-core/fs/zfs/zfs.c - `__. - This step creates a separate boot pool for ``/boot`` with the features - limited to only those that GRUB supports, allowing the root pool to use - any/all features. - - Features enabled with ``-o compatibility=grub2`` can be seen - `here `__. - -#. Create root pool:: - - zpool create \ - -o ashift=12 \ - -o autotrim=on \ - -R /mnt \ - -O acltype=posixacl \ - -O canmount=off \ - -O compression=zstd \ - -O dnodesize=auto \ - -O normalization=formD \ - -O relatime=on \ - -O xattr=sa \ - -O mountpoint=/ \ - rpool_$INST_UUID \ - $INST_VDEV \ - $(for i in ${DISK}; do - printf "$i-part3 "; - done) - - **Notes:** - - - The use of ``ashift=12`` is recommended here because many drives - today have 4 KiB (or larger) physical sectors, even though they - present 512 B logical sectors. Also, a future replacement drive may - have 4 KiB physical sectors (in which case ``ashift=12`` is desirable) - or 4 KiB logical sectors (in which case ``ashift=12`` is required). - - Setting ``-O acltype=posixacl`` enables POSIX ACLs globally. If you - do not want this, remove that option, but later add - ``-o acltype=posixacl`` (note: lowercase “o”) to the ``zfs create`` - for ``/var/log``, as `journald requires ACLs - `__ - - Setting ``normalization=formD`` eliminates some corner cases relating - to UTF-8 filename normalization. It also implies ``utf8only=on``, - which means that only UTF-8 filenames are allowed. If you care to - support non-UTF-8 filenames, do not use this option. For a discussion - of why requiring UTF-8 filenames may be a bad idea, see `The problems - with enforced UTF-8 only filenames - `__. - - ``recordsize`` is unset (leaving it at the default of 128 KiB). If you - want to tune it (e.g. ``-o recordsize=1M``), see `these - `__ `various - `__ `blog - `__ - `posts - `__. - - Setting ``relatime=on`` is a middle ground between classic POSIX - ``atime`` behavior (with its significant performance impact) and - ``atime=off`` (which provides the best performance by completely - disabling atime updates). Since Linux 2.6.30, ``relatime`` has been - the default for other filesystems. See `RedHat’s documentation - `__ - for further information. - - Setting ``xattr=sa`` `vastly improves the performance of extended - attributes - `__. - Inside ZFS, extended attributes are used to implement POSIX ACLs. - Extended attributes can also be used by user-space applications. - `They are used by some desktop GUI applications. - `__ - `They can be used by Samba to store Windows ACLs and DOS attributes; - they are required for a Samba Active Directory domain controller. - `__ - Note that ``xattr=sa`` is `Linux-specific - `__. If you move your - ``xattr=sa`` pool to another OpenZFS implementation besides ZFS-on-Linux, - extended attributes will not be readable (though your data will be). If - portability of extended attributes is important to you, omit the - ``-O xattr=sa`` above. Even if you do not want ``xattr=sa`` for the whole - pool, it is probably fine to use it for ``/var/log``. - - Make sure to include the ``-part3`` portion of the drive path. If you - forget that, you are specifying the whole disk, which ZFS will then - re-partition, and you will lose the bootloader partition(s). - -#. This section implements dataset layout as described in `overview <0-overview.html>`__. - - Create root system container: - - - Unencrypted:: - - zfs create \ - -o canmount=off \ - -o mountpoint=none \ - rpool_$INST_UUID/$INST_ID - - - Encrypted: - - Pick a strong password. Once compromised, changing password will not keep your - data safe. See ``zfs-change-key(8)`` for more info:: - - zfs create \ - -o canmount=off \ - -o mountpoint=none \ - -o encryption=on \ - -o keylocation=prompt \ - -o keyformat=passphrase \ - rpool_$INST_UUID/$INST_ID - - Create other system datasets:: - - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID - zfs create -o canmount=off -o mountpoint=none bpool_$INST_UUID/$INST_ID/BOOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/ROOT - zfs create -o canmount=off -o mountpoint=none rpool_$INST_UUID/$INST_ID/DATA - zfs create -o mountpoint=/boot -o canmount=noauto bpool_$INST_UUID/$INST_ID/BOOT/default - zfs create -o mountpoint=/ -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default - zfs create -o mountpoint=/ -o canmount=noauto rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/default - zfs mount bpool_$INST_UUID/$INST_ID/BOOT/default - for i in {usr,var,var/lib}; - do - zfs create -o canmount=off rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - for i in {home,root,srv,usr/local,var/log,var/spool}; - do - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/$i - done - chmod 750 /mnt/root - -#. Format and mount ESP:: - - for i in ${DISK}; do - mkfs.vfat -n EFI ${i}-part1 - mkdir -p /mnt/boot/efis/${i##*/}-part1 - mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1 - done - - mkdir -p /mnt/boot/efi - mount -t vfat ${INST_PRIMARY_DISK}-part1 /mnt/boot/efi - -#. Create separate user dataset at ``/home/User``, dateset name can be - changed later:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/home/User - - If needed, snapshot, rollback and other related permissions can be - delegated to the user later. - -#. Create optional program data datasets to omit data from rollback:: - - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/games - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/www - # for GNOME - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/AccountsService - # for Docker - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/docker - # for NFS - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/nfs - # for LXC - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/lxc - # for LibVirt - zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/libvirt - ##other application - # zfs create -o canmount=on rpool_$INST_UUID/$INST_ID/DATA/default/var/lib/$name - - Add other datasets when needed, such as PostgreSQL. - -#. Install base packages:: - - dnf --installroot=/mnt --releasever=${INST_RHEL_VER} -y install \ - ${RHEL_ZFS_REPO} @core epel-release grub2-efi-x64 grub2-pc-modules \ - grub2-efi-x64-modules shim-x64 efibootmgr \ - kernel kernel-devel python3-dnf-plugin-post-transaction-actions - dnf config-manager --installroot=/mnt --disable zfs - dnf config-manager --installroot=/mnt --enable zfs-kmod - dnf install --installroot=/mnt -y zfs zfs-dracut - -#. Update zfs repo if a newer release is available:: - - source /mnt/etc/os-release - RHEL_ZFS_REPO_NEW=https://zfsonlinux.org/epel/zfs-release.el${VERSION_ID/./_}.noarch.rpm - dnf install --installroot=/mnt -y $RHEL_ZFS_REPO_NEW || true - -#. Optional: enable boot environment support and dnf integration:: - - dnf --installroot=/mnt copr enable -y m0p/bieaz - dnf --installroot=/mnt install -y bieaz python3-dnf-plugin-rozb3 - - If multi-disk setup is used, enable multi-disk - support inside ``/mnt/etc/bieaz.cfg``. diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst deleted file mode 100644 index d9848e2a3..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/3-system-configuration.rst +++ /dev/null @@ -1,103 +0,0 @@ -.. highlight:: sh - -System Configuration -====================== - -.. contents:: Table of Contents - :local: - -#. Generate fstab:: - - genfstab -U /mnt | sed 's;zfs[[:space:]]*;zfs zfsutil,;g' | grep "zfs zfsutil" >> /mnt/etc/fstab - for i in ${DISK}; do - echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab - done - echo UUID=$(blkid -s UUID -o value ${INST_PRIMARY_DISK}-part1) /boot/efi vfat \ - x-systemd.idle-timeout=1min,x-systemd.automount,noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab - if [ "${INST_PARTSIZE_SWAP}" != "" ]; then - for i in ${DISK}; do - echo ${i##*/}-part4-swap ${i}-part4 /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256,discard >> /mnt/etc/crypttab - echo /dev/mapper/${i##*/}-part4-swap none swap x-systemd.requires=cryptsetup.target,defaults 0 0 >> /mnt/etc/fstab - done - fi - - By default, systemd will halt boot process if any entry in ``/etc/fstab`` fails - to mount. This is unnecessary for mirrored EFI boot partitions. - With the above mount options, systemd will skip mounting them at boot, - only mount them on demand when accessed. - -#. Configure dracut:: - - echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf - -#. Force load mpt3sas module if used:: - - if grep mpt3sas /proc/modules; then - echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf - fi - -#. Interactively set locale, keymap, timezone, hostname and root password:: - - rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD - - This can be non-interactive, see man page for details:: - - rm -f /mnt/etc/localtime - systemd-firstboot --root=/mnt \ - --locale="en_US.UTF-8" --locale-messages="en_US.UTF-8" \ - --keymap=us --timezone="Europe/Berlin" --hostname=myHost \ - --root-password=PASSWORD - - ``systemd-firstboot`` have bugs, root password is set below. - -#. Generate host id:: - - zgenhostid -f -o /mnt/etc/hostid - -#. Install locale package, example for English locale:: - - dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en - - Program will show errors if not installed. - -#. Enable ZFS services:: - - systemctl enable zfs-import-scan.service zfs-import.target zfs-zed zfs.target --root=/mnt - systemctl disable zfs-mount --root=/mnt - - At boot, datasets on rpool are mounted with ``/etc/fstab``, - which can control the mounting process more precisely than ``zfs-mount.service``. - -#. By default SSH server is enabled, allowing root login by password, - disable SSH server:: - - systemctl disable sshd --root=/mnt - systemctl enable firewalld --root=/mnt - -#. Chroot:: - - echo "INST_PRIMARY_DISK=$INST_PRIMARY_DISK - INST_LINVAR=$INST_LINVAR - INST_UUID=$INST_UUID - INST_ID=$INST_ID - unalias -a - TERM=xterm - INST_VDEV=$INST_VDEV - INST_VDEV=$INST_VDEV - DISK=\"$DISK\"" > /mnt/root/chroot - history -w /mnt/home/sys-install-pre-chroot.txt - arch-chroot /mnt bash --login - -#. Source variables:: - - source /root/chroot - -#. For SELinux, relabel filesystem on reboot:: - - fixfiles -F onboot - -#. Set root password:: - - passwd diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/5-bootloader.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/5-bootloader.rst deleted file mode 100644 index 81b5fb8c9..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/5-bootloader.rst +++ /dev/null @@ -1,258 +0,0 @@ -.. highlight:: sh - -Bootloader -====================== - -.. contents:: Table of Contents - :local: - -Apply workarounds -~~~~~~~~~~~~~~~~~~~~ -Currently GRUB has multiple compatibility problems with ZFS, -especially with regards to newer ZFS features. -Workarounds have to be applied. - -#. grub2-probe fails to get canonical path - - When persistent device names ``/dev/disk/by-id/*`` are used - with ZFS, GRUB will fail to resolve the path of the boot pool - device. Error:: - - # /usr/bin/grub2-probe: error: failed to get canonical path of `/dev/virtio-pci-0000:06:00.0-part3'. - - Solution:: - - echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh - source /etc/profile.d/zpool_vdev_name_path.sh - -#. Pool name missing - - See `this bug report `__. - Root pool name is missing from ``root=ZFS=rpool_$INST_UUID/ROOT/default`` - kernel cmdline in generated ``grub.cfg`` file. - - A workaround is to replace the pool name detection with ``zdb`` - command:: - - sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux - -Install GRUB -~~~~~~~~~~~~~~~~~~~~ - -#. If using virtio disk, add driver to initrd:: - - echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf - -#. Generate initrd:: - - rm -f /etc/zfs/zpool.cache - touch /etc/zfs/zpool.cache - chmod a-w /etc/zfs/zpool.cache - chattr +i /etc/zfs/zpool.cache - for directory in /lib/modules/*; do - kernel_version=$(basename $directory) - dracut --force --kver $kernel_version - done - -#. Load ZFS modules and disable BLS:: - - echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub - -#. Create GRUB boot directory, in ESP and boot pool:: - - mkdir -p /boot/efi/EFI/rocky # EFI GRUB dir - mkdir -p /boot/efi/EFI/rocky/grub2 # legacy GRUB dir - mkdir -p /boot/grub2 - - Boot environment-specific configuration (kernel, etc) - is stored in ``/boot/grub2/grub.cfg``, enabling rollback. - -#. When in doubt, install both legacy boot - and EFI. - -#. If using legacy booting, install GRUB to every disk:: - - for i in ${DISK}; do - grub2-install --boot-directory /boot/efi/EFI/rocky --target=i386-pc $i - done - -#. If using EFI:: - - for i in ${DISK}; do - efibootmgr -cgp 1 -l "\EFI\rocky\shimx64.efi" \ - -L "rocky-${i##*/}" -d ${i} - done - cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/rocky - -#. Generate GRUB Menu: - - Apply workaround:: - - tee /etc/grub.d/09_fix_root_on_zfs <> /etc/default/grub - - - ``/usr/sbin/grub2-probe: error: ../grub-core/kern/fs.c:120:unknown filesystem.`` - This is fixed by /etc/grub.d/09_fix_root_on_zfs - -#. For both legacy and EFI booting: mirror ESP content:: - - ESP_MIRROR=$(mktemp -d) - unalias -a - cp -r /boot/efi/EFI $ESP_MIRROR - for i in /boot/efis/*; do - cp -r $ESP_MIRROR/EFI $i - done - -#. Automatically regenerate GRUB menu on kernel update:: - - tee /etc/dnf/plugins/post-transaction-actions.d/00-update-grub-menu-for-kernel.action </dev/null - # kernel-core package contains vmlinuz and initramfs - # change package name if non-standard kernel is used - kernel-core:in:/usr/local/sbin/update-grub-menu.sh - kernel-core:out:/usr/local/sbin/update-grub-menu.sh - EOF - - tee /usr/local/sbin/update-grub-menu.sh <<-'EOF' >/dev/null - #!/bin/sh - export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin - export ZPOOL_VDEV_NAME_PATH=YES - source /etc/os-release - grub2-mkconfig -o /boot/efi/EFI/${ID}/grub.cfg - cp /boot/efi/EFI/${ID}/grub.cfg /boot/efi/EFI/${ID}/grub2/grub.cfg - cp /boot/efi/EFI/${ID}/grub.cfg /boot/grub2/grub.cfg - ESP_MIRROR=$(mktemp -d) - cp -r /boot/efi/EFI $ESP_MIRROR - for i in /boot/efis/*; do - cp -r $ESP_MIRROR/EFI $i - done - rm -rf $ESP_MIRROR - EOF - - chmod +x /usr/local/sbin/update-grub-menu.sh - -#. Notes for GRUB on RHEL - - To support Secure Boot, GRUB has been heavily modified by Fedora, - namely: - - - ``grub2-install`` is `disabled for UEFI `__ - - Only a static, signed version of bootloader is copied to EFI system partition - - This signed bootloader does not have built-in support for either ZFS or LUKS containers - - This signed bootloader only loads configuration from ``/boot/efi/EFI/fedora/grub.cfg`` - - Unrelated to Secure Boot, GRUB has also been modified to provide optional - support for `systemd bootloader specification (bls) `__. - Currently ``blscfg.mod`` is incompatible with root on ZFS. - - As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade. - Or else the new kernel will not be recognized and system will boot the old kernel - on reboot. - - Also see `Fedora docs for GRUB - `__. - -Finish Installation -~~~~~~~~~~~~~~~~~~~~ - -#. Exit chroot:: - - exit - -#. Take a snapshot of the clean installation for future use:: - - zfs snapshot -r rpool_$INST_UUID/$INST_ID@install - zfs snapshot -r bpool_$INST_UUID/$INST_ID@install - -#. Unmount EFI system partition:: - - umount /mnt/boot/efi - umount /mnt/boot/efis/* - -#. Export pools:: - - zpool export bpool_$INST_UUID - zpool export rpool_$INST_UUID - -#. Reboot:: - - reboot - -Post installaion -~~~~~~~~~~~~~~~~ - -#. If you have other data pools, generate list of datasets for `zfs-mount-generator - `__ to mount them at boot:: - - DATA_POOL='tank0 tank1' - - # tab-separated zfs properties - # see /etc/zfs/zed.d/history_event-zfs-list-cacher.sh - export \ - PROPS="name,mountpoint,canmount,atime,relatime,devices,exec\ - ,readonly,setuid,nbmand,encroot,keylocation" - - for i in $DATA_POOL; do - zfs list -H -t filesystem -o $PROPS -r $i > /etc/zfs/zfs-list.cache/$i - done - -#. After reboot, consider adding a normal user:: - - # with root permissions - sudo -i - - # store user name in a variable - myUser=UserName - - # rename default `User` to new user name - zfs rename $(df --output=source /home | tail -n +2)/User $(df --output=source /home | tail -n +2)/${myUser} - - # update entry in fstab - sed -i "s|/home/User|/home/${myUser}|g" /etc/fstab - - # add user - useradd --no-create-home --user-group --home-dir /home/${myUser} --comment 'My Name' ${myUser} - - # delegate snapshot and destroy permissions of the home dataset to - # new user - zfs allow -u ${myUser} mount,snapshot,destroy $(df --output=source /home | tail -n +2)/${myUser} - - # fix permissions - chown --recursive ${myUser}:${myUser} /home/${myUser} - chmod 700 /home/${myUser} - - # fix selinux context - restorecon /home/${myUser} - - # set new password for user - passwd ${myUser} - - Set up cron job to snapshot user home everyday:: - - dnf install cronie - systemctl enable --now crond - crontab -eu ${myUser} - #@daily zfs snap $(df --output=source /home/${myUser} | tail -n +2)@$(dd if=/dev/urandom of=/dev/stdout bs=1 count=100 2>/dev/null |tr -dc 'a-z0-9' | cut -c-6) - zfs list -t snapshot -S creation $(df --output=source /home/${myUser} | tail -n +2) - - Install package groups:: - - dnf group list --hidden -v # query package groups - dnf group install 'Virtualization Host' diff --git a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/6-recovery.rst b/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/6-recovery.rst deleted file mode 100644 index 1dad98b27..000000000 --- a/docs/Getting Started/RHEL-based distro/RHEL 8-based distro Root on ZFS/6-recovery.rst +++ /dev/null @@ -1,211 +0,0 @@ -.. highlight:: sh - -Recovery -====================== - -.. contents:: Table of Contents - :local: - -GRUB Tips -------------- - -Boot from GRUB rescue -~~~~~~~~~~~~~~~~~~~~~~~ - -If bootloader file is damaged, it's still possible -to boot computer with GRUB rescue image. - -This section is also applicable if you are in -``grub rescue>``. - -#. On another computer, generate rescue image with:: - - pacman -S --needed mtools libisoburn grub - grub-install - grub-mkrescue -o grub-rescue.img - dd if=grub-rescue.img of=/dev/your-usb-stick - - Boot computer from the rescue media. - Both legacy and EFI mode are supported. - - Or `download generated GRUB rescue image `__. - -#. List available disks with ``ls`` command:: - - grub> ls (hd # press tab - Possible devices are: - - hd0 hd1 hd2 hd3 - -#. List partitions by pressing tab key: - - .. code-block:: text - - grub> ls (hd0 # press tab - Possible partitions are: - - Device hd0: No known filesystem detected - Sector size 512B - Total size 20971520KiB - Partition hd0,gpt1: Filesystem type fat - Label `EFI', UUID 0DF5-3A76 - Partition start at 1024KiB - Total size 1048576KiB - Partition hd0,gpt2: No known filesystem detected - Partition start at 1049600KiB - Total size 4194304KiB - - - If boot pool is encrypted: - - Unlock it with ``cryptomount``:: - - grub> insmod luks - grub> cryptomount hd0,gpt2 - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (af5a240e13e24483acf02600d61e0f36): - Slot 1 opened - - Unlocked LUKS container is ``(crypto0)``: - - .. code-block:: text - - grub> ls (crypto0) - Device crypto0: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - - - If boot pool is not encrypted: - - .. code-block:: text - - grub> ls (hd0,gpt2) - Device hd0,gpt2: Filesystem type zfs - Label `bpool_ip3tdb' - Last modification - time 2021-05-03 12:14:08 Monday, UUID f14d7bdf89fe21fb - Sector size 512B - - Total size 4192256KiB - -#. List boot environments nested inside ``bpool/$INST_ID/BOOT``:: - - grub> ls (crypto0)/sys/BOOT - @/ default/ be0/ - -#. Instruct GRUB to load configuration from ``be0`` boot environment:: - - grub> prefix=(crypto0)/sys/BOOT/be0/@/grub - grub> configfile $prefix/grub.cfg - -#. GRUB menu should now appear. - -#. After entering system, `reinstall GRUB <#grub-installation>`__. - -Switch GRUB prefix when disk fails -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you are using LUKS encrypted boot pool with multiple disks, -the primary disk failed, GRUB will fail to load configuration. - -If there's still enough redundancy for the boot pool, try fix -GRUB with the following method: - -#. Ensure ``Slot 1 opened`` message - is shown - - .. code-block:: text - - Welcome to GRUB! - - error: no such cryptodisk found. - Attempting to decrypt master key... - Enter passphrase for hd0,gpt2 (c0987ea1a51049e9b3056622804de62a): - Slot 1 opened - error: disk `cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf' not found. - Entering rescue mode... - grub rescue> - - If ``error: access denied.`` is shown, - try re-enter password with:: - - grub rescue> cryptomount hd0,gpt2 - -#. Check prefix:: - - grub rescue > set - # prefix=(cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf)/sys/BOOT/be0@/grub - # root=cryptouuid/47ed1b7eb0014bc9a70aede3d8714faf - -#. Set correct ``prefix`` and ``root`` by replacing - ``cryptouuid/UUID`` with ``crypto0``:: - - grub rescue> prefix=(crypto0)/sys/BOOT/default@/grub - grub rescue> root=crypto0 - -#. Boot GRUB:: - - grub rescue> insmod normal - grub rescue> normal - - GRUB should then boot normally. - -#. After entering system, edit ``/etc/fstab`` to promote - one backup to ``/boot/efi``. - -#. Make the change to ``prefix`` and ``root`` - permanent by `reinstalling GRUB <#grub-installation>`__. - -Access system in chroot ------------------------ - -#. Go through `preparation <1-preparation.html>`__. - -#. Import and unlock root and boot pool:: - - zpool import -NR /mnt rpool_$INST_UUID - zpool import -NR /mnt bpool_$INST_UUID - - If using password:: - - zfs load-key rpool_$INST_UUID/$INST_ID - - If using keyfile:: - - zfs load-key -L file:///path/to/keyfile rpool_$INST_UUID/$INST_ID - -#. Find the current boot environment:: - - zfs list - BE=default - -#. Mount root filesystem:: - - zfs mount rpool_$INST_UUID/$INST_ID/ROOT/$BE - -#. chroot into the system:: - - arch-chroot /mnt /bin/bash --login - zfs mount -a - mount -a - -#. Finish rescue. See `finish installation <#finish-installation>`__. - -Backup and migrate existing installation ----------------------------------------- -With the help of `zfs send -`__ -it is relatively easy to perform a system backup and migration. - -#. Create a snapshot of root file system:: - - zfs snapshot -r rpool/$INST_ID@backup - zfs snapshot -r bpool/$INST_ID@backup - -#. Save snapshot to a file or pipe to SSH:: - - zfs send --options rpool/$INST_ID@backup > /backup/$INST_ID-rpool - zfs send --options bpool/$INST_ID@backup > /backup/$INST_ID-bpool - -#. Re-create partitions and root/boot - pool on target system. - -#. Restore backup:: - - zfs recv rpool_new/$INST_ID < /backup/$INST_ID-rpool - zfs recv bpool_new/$INST_ID < /backup/$INST_ID-bpool - -#. Chroot and reinstall bootloader. - -#. Update pool name in ``/etc/fstab``, ``/boot/grub/grub.cfg`` - and ``/etc/zfs/zfs-list.cache/*``. - -#. Update device name, etc, in ``/etc/fstab`` and ``/etc/crypttab``. diff --git a/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS.rst b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS.rst new file mode 100644 index 000000000..f55a18441 --- /dev/null +++ b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS.rst @@ -0,0 +1,11 @@ +RHEL Root on ZFS +======================================= +`Start here `__. + +Contents +-------- +.. toctree:: + :maxdepth: 2 + :glob: + + RHEL-based distro Root on ZFS/* diff --git a/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/1-preparation.rst b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/1-preparation.rst new file mode 100644 index 000000000..58cad9f72 --- /dev/null +++ b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/1-preparation.rst @@ -0,0 +1,78 @@ +.. highlight:: sh + +Preparation +====================== + +.. contents:: Table of Contents + :local: + +#. Disable Secure Boot. ZFS modules can not be loaded if Secure Boot is enabled. +#. Download a variant of `AlmaLinux Minimal Live ISO + `__ and boot from it. +#. Connect to the Internet. +#. Set root password or ``/root/.ssh/authorized_keys``. +#. Start SSH server:: + + echo PermitRootLogin yes >> /etc/ssh/sshd_config + systemctl restart sshd + +#. Connect from another computer:: + + ssh root@192.168.1.19 + +#. Target disk + + List available disks with:: + + ls /dev/disk/by-id/* + + If using virtio as disk bus, use ``/dev/disk/by-path/*``. + + Declare disk array:: + + DISK='/dev/disk/by-id/ata-FOO /dev/disk/by-id/nvme-BAR' + + For single disk installation, use:: + + DISK='/dev/disk/by-id/disk1' + +#. Set partition size: + + Set swap size. It's `recommended `__ + to setup a swap partition. If you intend to use hibernation, + the minimum should be no less than RAM size. Skip if swap is not needed:: + + INST_PARTSIZE_SWAP=8 + + Root pool size, use all remaining disk space if not set:: + + INST_PARTSIZE_RPOOL= + +#. Temporarily set SELinux to permissive in live environment:: + + setenforce 0 + + SELinux will be enabled on the installed system. + +#. Add ZFS repo:: + + dnf install -y https://zfsonlinux.org/epel/zfs-release-el-2-1.noarch.rpm + +#. Check available repos:: + + dnf repolist --all + +#. Install ZFS packages:: + + dnf config-manager --disable zfs + dnf config-manager --enable zfs-kmod + dnf install -y zfs + # if gpg import fails, add --nogpgcheck + +#. Load kernel modules:: + + modprobe zfs + +#. Install partition tool:: + + dnf install -y gdisk dosfstools diff --git a/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/2-system-installation.rst b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/2-system-installation.rst new file mode 100644 index 000000000..525d3ddee --- /dev/null +++ b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/2-system-installation.rst @@ -0,0 +1,146 @@ +.. highlight:: sh + +System Installation +====================== + +.. contents:: Table of Contents + :local: + +#. Partition the disks:: + + for i in ${DISK}; do + + sgdisk --zap-all $i + + sgdisk -n1:1M:+1G -t1:EF00 $i + + sgdisk -n2:0:+4G -t2:BE00 $i + + test -z $INST_PARTSIZE_SWAP || sgdisk -n4:0:+${INST_PARTSIZE_SWAP}G -t4:8200 $i + + if test -z $INST_PARTSIZE_RPOOL; then + sgdisk -n3:0:0 -t3:BF00 $i + else + sgdisk -n3:0:+${INST_PARTSIZE_RPOOL}G -t3:BF00 $i + fi + + sgdisk -a1 -n5:24K:+1000K -t5:EF02 $i + done + +#. Create boot pool:: + + zpool create \ + -o compatibility=grub2 \ + -o ashift=12 \ + -o autotrim=on \ + -O acltype=posixacl \ + -O canmount=off \ + -O compression=lz4 \ + -O devices=off \ + -O normalization=formD \ + -O relatime=on \ + -O xattr=sa \ + -O mountpoint=/boot \ + -R /mnt \ + bpool \ + mirror \ + $(for i in ${DISK}; do + printf "$i-part2 "; + done) + + If not using a multi-disk setup, remove ``mirror``. + + You should not need to customize any of the options for the boot pool. + + GRUB does not support all of the zpool features. See ``spa_feature_names`` + in `grub-core/fs/zfs/zfs.c + `__. + This step creates a separate boot pool for ``/boot`` with the features + limited to only those that GRUB supports, allowing the root pool to use + any/all features. + + Features enabled with ``-o compatibility=grub2`` can be seen + `here `__. + +#. Create root pool:: + + zpool create \ + -o ashift=12 \ + -o autotrim=on \ + -R /mnt \ + -O acltype=posixacl \ + -O canmount=off \ + -O compression=zstd \ + -O dnodesize=auto \ + -O normalization=formD \ + -O relatime=on \ + -O xattr=sa \ + -O mountpoint=/ \ + rpool \ + mirror \ + $(for i in ${DISK}; do + printf "$i-part3 "; + done) + + If not using a multi-disk setup, remove ``mirror``. + +#. This section implements dataset layout as described in `overview <1-preparation.html>`__. + + Create root system container: + + - Unencrypted:: + + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + rpool/redhat + + - Encrypted: + + Pick a strong password. Once compromised, changing password will not keep your + data safe. See ``zfs-change-key(8)`` for more info:: + + zfs create \ + -o canmount=off \ + -o mountpoint=none \ + -o encryption=on \ + -o keylocation=prompt \ + -o keyformat=passphrase \ + rpool/redhat + + Create system datasets:: + + zfs create -o canmount=on -o mountpoint=/ rpool/redhat/root + zfs create -o canmount=on -o mountpoint=/home rpool/redhat/home + zfs create -o canmount=off -o mountpoint=/var rpool/redhat/var + zfs create -o canmount=on rpool/redhat/var/lib + zfs create -o canmount=on rpool/redhat/var/log + + Create boot dataset:: + + zfs create -o canmount=on -o mountpoint=/boot bpool/redhat + +#. Format and mount ESP:: + + for i in ${DISK}; do + mkfs.vfat -n EFI ${i}-part1 + mkdir -p /mnt/boot/efis/${i##*/}-part1 + mount -t vfat ${i}-part1 /mnt/boot/efis/${i##*/}-part1 + done + + mkdir -p /mnt/boot/efi + mount -t vfat $(echo $DISK | cut -f1 -d\ )-part1 /mnt/boot/efi + +#. Install packages:: + + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \ + @core grub2-efi-x64 grub2-pc-modules grub2-efi-x64-modules shim-x64 efibootmgr kernel + + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) -y install \ + https://zfsonlinux.org/epel/zfs-release-el-2-1.noarch.rpm + + dnf config-manager --installroot=/mnt --disable zfs + dnf config-manager --installroot=/mnt --enable zfs-kmod + + dnf --installroot=/mnt --releasever=$(source /etc/os-release ; echo $VERSION_ID) \ + --nogpgcheck -y install zfs zfs-dracut diff --git a/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/3-system-configuration.rst b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/3-system-configuration.rst new file mode 100644 index 000000000..94d60db93 --- /dev/null +++ b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/3-system-configuration.rst @@ -0,0 +1,66 @@ +.. highlight:: sh + +System Configuration +====================== + +.. contents:: Table of Contents + :local: + +#. Generate fstab:: + + mkdir -p /mnt/etc/ + for i in ${DISK}; do + echo UUID=$(blkid -s UUID -o value ${i}-part1) /boot/efis/${i##*/}-part1 vfat \ + umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab + done + echo $(echo $DISK | cut -f1 -d\ )-part1 /boot/efi vfat \ + noauto,umask=0022,fmask=0022,dmask=0022 0 1 >> /mnt/etc/fstab + +#. Configure dracut:: + + echo 'add_dracutmodules+=" zfs "' > /mnt/etc/dracut.conf.d/zfs.conf + +#. Force load mpt3sas module if used:: + + if grep mpt3sas /proc/modules; then + echo 'forced_drivers+=" mpt3sas "' >> /mnt/etc/dracut.conf.d/zfs.conf + fi + +#. Set locale, keymap, timezone, hostname and root password:: + + rm -f /mnt/etc/localtime + systemd-firstboot --root=/mnt --prompt --root-password=PASSWORD --force + +#. Generate host id:: + + zgenhostid -f -o /mnt/etc/hostid + +#. Install locale package, example for English locale:: + + dnf --installroot=/mnt install -y glibc-minimal-langpack glibc-langpack-en + +#. Enable ZFS services:: + + systemctl enable zfs-import-scan.service zfs-mount zfs-import.target zfs-zed zfs.target --root=/mnt + +#. By default SSH server is enabled, allowing root login by password, + disable SSH server:: + + systemctl disable sshd --root=/mnt + systemctl enable firewalld --root=/mnt + +#. Chroot:: + + m='/dev /proc /sys' + for i in $m; do mount --rbind $i /mnt/$i; done + + history -w /mnt/home/sys-install-pre-chroot.txt + chroot /mnt /usr/bin/env DISK=$DISK bash --login + +#. For SELinux, relabel filesystem on reboot:: + + fixfiles -F onboot + +#. Set root password:: + + passwd diff --git a/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/5-bootloader.rst b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/5-bootloader.rst new file mode 100644 index 000000000..b7e307d3a --- /dev/null +++ b/docs/Getting Started/RHEL-based distro/RHEL-based distro Root on ZFS/5-bootloader.rst @@ -0,0 +1,129 @@ +.. highlight:: sh + +Bootloader +====================== + +.. contents:: Table of Contents + :local: + +Apply workarounds +~~~~~~~~~~~~~~~~~~~~ +Currently GRUB has multiple compatibility problems with ZFS, +especially with regards to newer ZFS features. +Workarounds have to be applied. + +#. grub2-probe fails to get canonical path + + When persistent device names ``/dev/disk/by-id/*`` are used + with ZFS, GRUB will fail to resolve the path of the boot pool + device. Error:: + + # /usr/bin/grub2-probe: error: failed to get canonical path of `/dev/virtio-pci-0000:06:00.0-part3'. + + Solution:: + + echo 'export ZPOOL_VDEV_NAME_PATH=YES' >> /etc/profile.d/zpool_vdev_name_path.sh + source /etc/profile.d/zpool_vdev_name_path.sh + +#. Pool name missing + + See `this bug report `__. + Root pool name is missing from ``root=ZFS=rpool_$INST_UUID/ROOT/default`` + kernel cmdline in generated ``grub.cfg`` file. + + A workaround is to replace the pool name detection with ``zdb`` + command:: + + sed -i "s|rpool=.*|rpool=\`zdb -l \${GRUB_DEVICE} \| grep -E '[[:blank:]]name' \| cut -d\\\' -f 2\`|" /etc/grub.d/10_linux + + Caution: this fix must be applied after every GRUB update and before generating the menu. + +Install GRUB +~~~~~~~~~~~~~~~~~~~~ + +#. If using virtio disk, add driver to initrd:: + + echo 'filesystems+=" virtio_blk "' >> /etc/dracut.conf.d/fs.conf + +#. Create empty cache file and generate initrd:: + + rm -f /etc/zfs/zpool.cache + touch /etc/zfs/zpool.cache + chmod a-w /etc/zfs/zpool.cache + chattr +i /etc/zfs/zpool.cache + + for directory in /lib/modules/*; do + kernel_version=$(basename $directory) + dracut --force --kver $kernel_version + done + +#. Load ZFS modules and disable BLS:: + + echo 'GRUB_ENABLE_BLSCFG=false' >> /etc/default/grub + +#. If using legacy booting, install GRUB to every disk:: + + for i in ${DISK}; do + grub2-install --target=i386-pc $i + done + +#. If using EFI:: + + for i in ${DISK}; do + efibootmgr -cgp 1 -l "\EFI\almalinux\shimx64.efi" \ + -L "almalinux-${i##*/}" -d ${i} + done + cp -r /usr/lib/grub/x86_64-efi/ /boot/efi/EFI/almalinux/ + +#. Generate GRUB Menu: + + Generate menu:: + + grub2-mkconfig -o /boot/grub2/grub.cfg + cp /boot/grub2/grub.cfg /boot/efi/EFI/almalinux/ + +#. For both legacy and EFI booting: mirror ESP content:: + + ESP_MIRROR=$(mktemp -d) + unalias -a + cp -r /boot/efi/EFI $ESP_MIRROR + for i in /boot/efis/*; do + cp -r $ESP_MIRROR/EFI $i + done + rm -rf $ESP_MIRROR + +#. Notes for GRUB on RHEL + + As bls is disabled, you will need to regenerate GRUB menu after each kernel upgrade. + Or else the new kernel will not be recognized and system will boot the old kernel + on reboot. + +Finish Installation +~~~~~~~~~~~~~~~~~~~~ + +#. Exit chroot:: + + exit + +#. Export pools:: + + umount -Rl /mnt + zpool export -a + +#. Reboot:: + + reboot + +#. On first reboot, the boot process will fail, with failure messages such + as "You are in Emergency Mode...Press Ctrl-D to continue". + + Wait for the computer to automatically reboot and the problem will be resolved. + +Post installaion +~~~~~~~~~~~~~~~~ +#. Install package groups:: + + dnf group list --hidden -v # query package groups + dnf group install @gnome-desktop + +#. Add new user, configure swap. diff --git a/docs/Getting Started/RHEL-based distro/index.rst b/docs/Getting Started/RHEL-based distro/index.rst index 3a24e01d2..ff32e7b56 100644 --- a/docs/Getting Started/RHEL-based distro/index.rst +++ b/docs/Getting Started/RHEL-based distro/index.rst @@ -146,15 +146,15 @@ And for RHEL/CentOS 8 and newer:: Use *zfs-testing* for DKMS packages and *zfs-testing-kmod* kABI-tracking kmod packages. -RHEL 8-based distro Root on ZFS +RHEL-based distro Root on ZFS ------------------------------- -`Start here `__. +`Start here `__. .. toctree:: :maxdepth: 1 :glob: - RHEL 8-based distro Root on ZFS/* + RHEL-based distro Root on ZFS/* .. _kABI-tracking kmod: https://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html .. _DKMS: https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support