docs: Document TPM-backed rootfs encryption

A long-requested feature is disk encryption. In the next Alpha we have
everything in place for TPM-backed disk encryption with
systemd-cryptenroll and Clevis, and with a network-backed disk secret
store with Tang.
Document the limitations, implications and the Ignition configs.

content/docs/latest/_index.md

@@ -117,6 +117,7 @@ more.
  * [iSCSI configuration][iscsi]
 
 #### Additional security options
+ * [Setting up LUKS disk encryption][luks-encryption]
  * [Customizing the SSH daemon][ssh-daemon]
  * [Configuring SSSD on Flatcar Container Linux][sssd-container-linux]
  * [Hardening a Flatcar Container Linux machine][hardening-container-linux]
@@ -215,6 +216,7 @@ Flatcar tutorial to deep dive into some Flatcar fundamental concepts.
 [update-conf]: setup/releases/update-conf
 [release-channels]: setup/releases/switching-channels
 [tasks-with-systemd]: setup/systemd/timers
+[luks-encryption]: setup/security/luks
 [ssh-daemon]: setup/security/customizing-sshd
 [sssd-container-linux]: setup/security/sssd
 [hardening-container-linux]: setup/security/hardening-guide

content/docs/latest/setup/security/luks.md

@@ -0,0 +1,245 @@
+---
+title: Setting up LUKS disk encryption
+linktitle: Setting up LUKS disk encryption
+description: Setting up LUKS disk encryption with automatic unlocking using systemd-cryptenroll or Clevis
+weight: 10
+---
+
+Depending on where you run Flatcar Container Linux you might want to protect the data on disk against attackers that could pull out a hard disk or get access to a snapshot copy of it.
+Like a laptop, a server can use disk encryption to protect the contents. However, since there is no-one to type the password for unlocking, the unlocking has to happen automatically. It's hard to do this in a secure way that also protects against attackers with prolonged physical access to the system or similar modification capabilities.
+
+TPM2 chips provide a method to unlock automatically because they can store a secret inside them and only reveal the secret based on a policy. Even a simple policy that always reveals the secret provides protection against many simple attacks such as stealing an encrypted hard disk without the computer's mainboard, or getting access to a disk snapshot.
+To prevent against attackers that can modify the system, the TPM policy must bind the secret against an exepected system state. Predicting this state is brittle and can lead to you either access yourself, or leaving a security hole open because the state did not capture all possible malicious modifications. A Linux system design that prevents against most malicious modifications without being too brittle is still a current development topic.
+
+TPM2-backed rootfs encryption in Flatcar is supported from version 3913.0.1 onwards. You can try it out with the `./flatcar_production_pxe.sh` Flatcar Qemu script through the `-T TPMSTATEDIR` flag.
+
+## Current limitations
+
+Flatcar Container Linux uses GRUB as bootloader which, on UEFI systems, can measure the system state based on the configuration code it executes. This means that the TPM PCR values would not be predictable/universal for a given Flatcar version but would have to be observed locally.
+Flatcar's use of the kernel cmdline for various OEM configurations and for triggering Ignition adds to that. This would mean that binding a secret against the full TPM PCR state only works realible when auto updates are disabled and when the first-boot setup or post-update GPT flag reading is not involved. However, the current GRUB setup in Flatcar does not measure anything on UEFI by default nor for BIOS systems, thus binding against the PCR values won't bind against the OS version. However, one can instruct GRUB to measure the boot code and the kernel. Also note that, as of now, Flatcar OEM extensions are also read from the OEM partition without creating measurements nor with pre-checking their checksum because the dm-verity protection policy for extension images is not flexible enough to allow users to also load unsigned extensions.
+
+## Simple disk encryption
+
+A simple way of protecting against limited attacks is to use the TPM as device-only secret store. A similar but slightly weaker security level can be achieved with a network secret store, such as [Tang][tang], that only works in the local network.
+
+Flatcar supports systemd-cryptenroll for TPM and Clevis for TPM or Tang secrets. The root disk and/or additional disks can be encrypted. Ignition/Butane has inbuilt Clevis support while for systemd-cryptenroll a helper service is needed as of now.
+
+By default systemd-cryptenroll will bind to PCR 7 which is the device firmware state. This is probably not so brittle but doesn't offer much protection anyway and can be disabled by explicitly setting an empty parameter value through `--tpm2-pcrs=` if it might cause disruption.
+
+Example for TPM2-backed rootfs encryption with systemd-cryptenroll (Butane YAML):
+
+```yaml
+variant: flatcar
+version: 1.0.0
+storage:
+  luks:
+  - name: rootencrypted
+    wipe_volume: true
+    device: "/dev/disk/by-partlabel/ROOT"
+  filesystems:
+    - device: /dev/mapper/rootencrypted
+      format: ext4
+      label: ROOT
+systemd:
+  units:
+    - name: cryptenroll-helper.service
+      enabled: true
+      contents: |
+        [Unit]
+        ConditionFirstBoot=true
+        OnFailure=emergency.target
+        OnFailureJobMode=isolate
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted --wipe-slot=0 --tpm2-pcrs= /dev/disk/by-partlabel/ROOT
+        ExecStart=rm /etc/luks/rootencrypted
+        [Install]
+        WantedBy=multi-user.target
+```
+
+For Clevis one needs a Butane version with support for the Flatcar 1.2 Butane spec (here using a dev version built from a [PR](https://github.com/coreos/butane/pull/523)).
+Example for TPM2-backed rootfs encryption with Clevis:
+
+```yaml
+variant: flatcar
+version: 1.2.0-experimental
+storage:
+  luks:
+  - name: rootencrypted
+    wipe_volume: true
+    device: "/dev/disk/by-partlabel/ROOT"
+    clevis:
+      tpm2: true
+  filesystems:
+    - device: /dev/mapper/rootencrypted
+      format: ext4
+      label: ROOT
+```
+
+Clevis can also use a Tang server as secret store (Butane YAML):
+
+```yaml
+variant: flatcar
+version: 1.2.0-experimental
+kernel_arguments:
+  should_exist:
+    - rd.networkd=1
+storage:
+  luks:
+  - name: rootencrypted
+    device: "/dev/disk/by-partlabel/ROOT"
+    clevis:
+      tang:
+        - url: "http://tang.mycompany.com"
+          thumbprint: "Hk1VN2eKhzaVqWhXtXxXIGF5LRZt4cBWWb07I1-a0NX"
+```
+
+## A more secure setup for disk encryption
+
+When disabling auto-updates and doing one additional reboot to get rid of the Ignition kernel cmdline parameter before binding against PCRs, one can use instruct GRUB to measure the kernel and boot code, and tell systemd-cryptenroll to bind against more than PCR 7. There are still limitations to the achieved protection because of the mentioned lack of extension signing on the OEM partition. Other issues may also reduce the security as this is an active area of development.
+
+Example for TPM2-backed rootfs encryption with systemd-cryptenroll and stronger PCR binding, including disabling updates (Butane YAML):
+
+```yaml
+variant: flatcar
+version: 1.0.0
+storage:
+  files:
+    - path: /oem/grub.cfg
+      append:
+        - inline: |
+            insmod tpm
+            tpm_record_pcrs 8-9
+    - path: /etc/flatcar/update.conf
+      overwrite: true
+      contents:
+        inline: |
+          SERVER=disabled
+  luks:
+  - name: rootencrypted
+    wipe_volume: true
+    device: "/dev/disk/by-partlabel/ROOT"
+  filesystems:
+    - device: /dev/mapper/rootencrypted
+      format: ext4
+      label: ROOT
+systemd:
+  units:
+    - name: cryptenroll-helper-first.service
+      enabled: true
+      contents: |
+        [Unit]
+        ConditionFirstBoot=true
+        OnFailure=emergency.target
+        OnFailureJobMode=isolate
+        After=first-boot-complete.target multi-user.target
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted --tpm2-pcrs= /dev/disk/by-partlabel/ROOT
+        ExecStart=mv /etc/luks/rootencrypted /etc/luks/rootencrypted-bind
+        ExecStart=sleep 10
+        ExecStart=systemctl reboot
+        [Install]
+        WantedBy=multi-user.target
+    - name: cryptenroll-helper-bind.service
+      enabled: true
+      contents: |
+        [Unit]
+        ConditionFirstBoot=false
+        ConditionPathExists=/etc/luks/rootencrypted-bind
+        OnFailure=emergency.target
+        OnFailureJobMode=isolate
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted-bind --tpm2-pcrs=4+5+7+8+9+11+12+13 --wipe-slot=tpm2 /dev/disk/by-partlabel/ROOT
+        ExecStart=mv /etc/luks/rootencrypted-bind /etc/luks/rootencrypted
+        [Install]
+        WantedBy=multi-user.target
+```
+
+Since we normally want updates you have to prevent locking yourself out by removing the PCR binding before updating and add it back after two reboots (a second reboot, after one minute of uptime in the updated system, is needed to create a stable GRUB code path where no GPT flags change). The following example shows how to automate this with a helper that runs systemd-cryptenroll as post-update hook.
+
+Example for TPM2-backed rootfs encryption with systemd-cryptenroll and stronger PCR binding, with a added unbinding while the update reboot is pending (Butane YAML):
+
+```yaml
+variant: flatcar
+version: 1.0.0
+storage:
+  files:
+    - path: /oem/grub.cfg
+      append:
+        - inline: |
+            insmod tpm
+            tpm_record_pcrs 8-9
+    - path: /oem/bin/oem-postinst
+      overwrite: true
+      mode: 0755
+      contents:
+        inline: |
+          #!/bin/bash
+          set -euo pipefail
+          # When the update fails to correctly apply, this runs again
+          if [ -e /etc/luks/rootencrypted-bound ]; then
+            mv /etc/luks/rootencrypted-bound /etc/luks/rootencrypted-bind
+          fi
+          # But since a reboot inbetween could have bound it again,
+          # remove the PCR binding for every run
+          systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted-bind --wipe-slot=tpm2 --tpm2-pcrs= /dev/disk/by-partlabel/ROOT
+  luks:
+  - name: rootencrypted
+    wipe_volume: true
+    device: "/dev/disk/by-partlabel/ROOT"
+  filesystems:
+    - device: /dev/mapper/rootencrypted
+      format: ext4
+      label: ROOT
+systemd:
+  units:
+    - name: cryptenroll-helper-first.service
+      enabled: true
+      contents: |
+        [Unit]
+        ConditionFirstBoot=true
+        OnFailure=emergency.target
+        OnFailureJobMode=isolate
+        After=first-boot-complete.target multi-user.target
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        ExecStart=systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted --tpm2-pcrs= /dev/disk/by-partlabel/ROOT
+        ExecStart=mv /etc/luks/rootencrypted /etc/luks/rootencrypted-bind
+        ExecStart=sleep 10
+        ExecStart=systemctl reboot
+        [Install]
+        WantedBy=multi-user.target
+    - name: cryptenroll-helper-bind.service
+      enabled: true
+      contents: |
+        [Unit]
+        ConditionFirstBoot=false
+        ConditionPathExists=/etc/luks/rootencrypted-bind
+        OnFailure=emergency.target
+        OnFailureJobMode=isolate
+        Before=update-engine.service
+        [Service]
+        Type=oneshot
+        RemainAfterExit=yes
+        # If we are in a newly updated /usr partition, force the new flag instead of waiting
+        # for update-engine, and directly do a reboot so that the GRUB code path will stabilize
+        # before we bind against the PCR values
+        ExecStart=bash -c 'set -euo pipefail; if ! cgpt show $(rootdev -s /usr) | grep "tries=0 successful=1" ; then flatcar-setgoodroot && echo Rebooting one more time after update && systemctl reboot && exit 1 ; fi'
+        ExecStart=systemd-cryptenroll --tpm2-device=auto --unlock-key-file=/etc/luks/rootencrypted-bind --tpm2-pcrs=4+5+7+8+9+11+12+13 --wipe-slot=tpm2 /dev/disk/by-partlabel/ROOT
+        ExecStart=mv /etc/luks/rootencrypted-bind /etc/luks/rootencrypted-bound
+        [Install]
+        WantedBy=multi-user.target
+```
+
+For debugging PCR value changes you can use `tpm2_pcrread` or `/usr/lib/systemd/systemd-pcrlock log`. Accessing single PCR values also works with `cat /sys/class/tpm/tpm0/pcr-sha256/NUMBER`.
+
+We hope to soon reduce the use of kernel parameters in Flatcar and change the GRUB code to stop reading GPT flags to make PCR binding more simple.
+In the future we hope to provide a Flatcar variant that can make use of systemd-boot for signed TPM policies that will allow to do auto updates and rollbacks without required re-enrollment, and make it possible to use systemd-pcrlock for additional control.
+
+[tang]: https://github.com/latchset/tang