Setting up crypted swap partition sometimes fails (racey)

[paulvt]

systemd version the issue has been seen with

237

Note: While we are aware that this version is over 2 versions old, we have noticed that no relevant changes have been made to the cryptsetup generator, a key part of this bug report, since this release (except for keydev support).

Used distribution

Ubuntu 18.04 LTS (Bionic)

Expected behaviour you didn't see

The crypted swap partition is set up correctly during all boot procedures on all our devices.

Unexpected behaviour you saw

The crypted swap partition set up succeeds but the device is not marked as active:

$ systemctl status dev-mapper-crypt_swap.device
● dev-mapper-crypt_swap.device
   Loaded: loaded
  Drop-In: /run/systemd/generator/dev-mapper-crypt_swap.device.d
           └─90-device-timeout.conf
   Active: inactive (dead)

Sep 26 11:52:22 hostname systemd[1]: dev-mapper-crypt_swap.device: Job dev-mapper-crypt_swap.device/start failed with result 'timeout'.
Sep 26 11:52:52 hostname systemd[1]: dev-mapper-crypt_swap.device: Job dev-mapper-crypt_swap.device/start timed out.
...etc

In case the swap partition is marked as that it should not fail, it blocks the boot process.

Steps to reproduce the problem

• Set up a crypted swap partition with the following fstab and crypttab entries:

/etc/fstab:

/dev/mapper/crypt_swap   none      swap sw 0 2

/etc/crypttab:

crypt_swap   PARTLABEL=SwapPartition   /dev/urandom swap,cipher=aes-xts-plain64

• Reboot the system

For us, the issue seems to cause problems where the system is either relatively slow CPU-wise or IO-wise.

Analysis

The cyptsetup-generate of systemd generates the following service file systemd-cryptsetup@crypt_swap.service:

# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
Conflicts=umount.target
IgnoreOnIsolate=true
After=cryptsetup-pre.target
Before=cryptsetup.target
After=systemd-random-seed.service
BindsTo=dev-disk-by\x2dpartlabel-SwapPartition.device
After=dev-disk-by\x2dpartlabel-SwapPartition.device
Before=umount.target
Before=dev-mapper-%i.swap

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
KeyringMode=shared
ExecStart=/lib/systemd/systemd-cryptsetup attach 'crypt_swap' '/dev/disk/by-partlabel/SwapPartition' '/dev/urandom' 'swap,cipher=aes-xts-plain64'
ExecStop=/lib/systemd/systemd-cryptsetup detach 'crypt_swap'
ExecStartPost=/sbin/mkswap '/dev/mapper/crypt_swap'

What we think that the following is going on:

• The systemd-cryptsetup attach call leads to systemd-udev creating the device /dev/mapper/crypt_setup with SYSTEMD_READY=0 as per /lib/udev/rules.d/99-systemd-rules and also starts setting up an inotify watch as per /lib/udev/rules.d/60-persistend-storage-dm.rules.
• On slow systems the setup of this watch races with the mkswap call that is done next.
  If mkswap is called before the watch is set up by udev, it misses the fact that the swap partition is now ready, the device keeps being marked as SYSTEMD_READY=0.
• As a result, the service dev-mapper-crypt_swap.device never becomes active and it blocks the boot because the job timeout is set to 0 for the service by a systemd override (in case the swap partition is considered "fail").

Below are the log files of the interaction of mkswap and systemd-udev for the cases where it goes right and wrong.

Correct

Sep 24 16:58:42 hostname systemd-cryptsetup[508]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev
...
Sep 24 16:58:46 hostname systemd-udevd[489]: handling device node '/dev/dm-1', devnum=b253:1, mode=0660, uid=0, gid=6
...
Sep 24 16:58:47 hostname systemd-udevd[485]: adding watch on '/dev/dm-1'
...
Sep 24 16:58:47 hostname mkswap[591]: Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
Sep 24 16:58:47 hostname mkswap[591]: no label, UUID=4a24083d-627c-416c-bbb0-aba431b9bfa5
...
Sep 24 16:58:47 hostname systemd-udevd[480]: inotify event: 8 for /dev/dm-1
...
Sep 24 16:58:47 hostname systemd-udevd[480]: device /dev/dm-1 closed, synthesising 'change'

Wrong

Sep 24 16:50:23 hostname systemd-cryptsetup[513]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev
...
Sep 24 16:50:27 hostname systemd-udevd[499]: handling device node '/dev/dm-1', devnum=b253:1, mode=0660, uid=0, gid=6
...
Sep 24 16:50:28 hostname mkswap[599]: Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
Sep 24 16:50:28 hostname mkswap[599]: no label, UUID=0a0ab441-8efe-44f3-92e4-2b8cb7a165c4
...
Sep 24 16:50:28 hostname systemd-udevd[498]: adding watch on '/dev/dm-1'

Considerations

While systemd often relies on proper notifications for other subsystems, such as LVM etc., in this case the setup is relatively bare and systemd itself calls the bare tool mkswap and relies on udev to mark the device as ready. We currently don't know how to solve this or get around the issue.

[arvidjaar]

creating the device /dev/mapper/crypt_setup with SYSTEMD_READY=0 as per /lib/udev/rules.d/99-systemd-rules

Wow! This is entirely bogus. There is no reason to prohibit use of device without recognizable format.

Actually #10154 is related. If systemd had ability to run commands to bring up device, this workaround would not be needed. It would just wait for mkswap to complete and proceed with any dependent services.

We currently don't know how to solve this or get around the issue.

Add udevadm trigger as one more ExecStartPost, after mkswap. This should reset SYSTEMD_READY.

[paulvt]

Indeed, I had already added the following override file systemd-cryptsetup@crypt_swap.service.d/90-trigger-udev.conf:

# Run udevadm trigger after the mkswap call in the original generated
# service

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

You seem to imply that the udev rules should not set SYSTEMD_READY=0 for a new crypt (opened) device, so changing that rule could be another workaround too?

[arvidjaar]

You seem to imply that the udev rules should not set SYSTEMD_READY=0 for a new crypt (opened) device, so changing that rule could be another workaround too?

Removing it now will break swap and tmp processing by starting followup units too early. This requires some non-trivial redesign of the whole mess.

[mmstick]

Indeed, I had already added the following override file systemd-cryptsetup@crypt_swap.service.d/90-trigger-udev.conf:

# Run udevadm trigger after the mkswap call in the original generated
# service

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

That seems to do the job. This is an issue experienced heavily on Pop!_OS, because we encrypt swap partitions by default. Most hardware is unaffected, but there are a few models that hit this error regularly. After applying this, the issue stopped entirely on the Galago Pro 2.

Now I need to find a way to apply this automatically for all installations.

[DemiMarie]

I strongly suspect that the real bug is in systemd-udevd: it emits an add event before establishing an inotify watch. As a result, it races against mkswap.

[ndusart]

Is there any plan to fix this ?

I'm experiencing this on Archlinux too. Most of the time, systemd will hang waiting for the swap partition while it is effectively created.

[grzegorzk]

I'm experiencing this issue too. Fresh Archlinux installs, boot hangs while waiting for crypt-swap device. I experience this issue 100% boots on 1CPU/1Gb RAM Linode.

On fresh install with systemd 244 I configured crypttab and fstab swap entries with noauto

Starting auto-generated swap unit manually works 100% of times.

systemd-networkd will also fail if cryp-swap is enabled. Without crypt-swap systemd-networkd will not fail.

Hope this helps!

Applying the workaround with udevadm trigger works. This seems to be the best solution for now.