Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting up crypted swap partition sometimes fails (racey) #10179

Open
paulvt opened this issue Sep 26, 2018 · 7 comments
Open

Setting up crypted swap partition sometimes fails (racey) #10179

paulvt opened this issue Sep 26, 2018 · 7 comments
Labels

Comments

@paulvt
Copy link

@paulvt paulvt commented Sep 26, 2018

systemd version the issue has been seen with

237

Note: While we are aware that this version is over 2 versions old, we have noticed that no relevant changes have been made to the cryptsetup generator, a key part of this bug report, since this release (except for keydev support).

Used distribution

Ubuntu 18.04 LTS (Bionic)

Expected behaviour you didn't see

The crypted swap partition is set up correctly during all boot procedures on all our devices.

Unexpected behaviour you saw

The crypted swap partition set up succeeds but the device is not marked as active:

$ systemctl status dev-mapper-crypt_swap.device
● dev-mapper-crypt_swap.device
   Loaded: loaded
  Drop-In: /run/systemd/generator/dev-mapper-crypt_swap.device.d
           └─90-device-timeout.conf
   Active: inactive (dead)

Sep 26 11:52:22 hostname systemd[1]: dev-mapper-crypt_swap.device: Job dev-mapper-crypt_swap.device/start failed with result 'timeout'.
Sep 26 11:52:52 hostname systemd[1]: dev-mapper-crypt_swap.device: Job dev-mapper-crypt_swap.device/start timed out.
...etc

In case the swap partition is marked as that it should not fail, it blocks the boot process.

Steps to reproduce the problem

  • Set up a crypted swap partition with the following fstab and crypttab entries:

/etc/fstab:

/dev/mapper/crypt_swap   none      swap sw 0 2

/etc/crypttab:

crypt_swap   PARTLABEL=SwapPartition   /dev/urandom swap,cipher=aes-xts-plain64
  • Reboot the system

For us, the issue seems to cause problems where the system is either relatively slow CPU-wise or IO-wise.

Analysis

The cyptsetup-generate of systemd generates the following service file systemd-cryptsetup@crypt_swap.service:

# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
Conflicts=umount.target
IgnoreOnIsolate=true
After=cryptsetup-pre.target
Before=cryptsetup.target
After=systemd-random-seed.service
BindsTo=dev-disk-by\x2dpartlabel-SwapPartition.device
After=dev-disk-by\x2dpartlabel-SwapPartition.device
Before=umount.target
Before=dev-mapper-%i.swap

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
KeyringMode=shared
ExecStart=/lib/systemd/systemd-cryptsetup attach 'crypt_swap' '/dev/disk/by-partlabel/SwapPartition' '/dev/urandom' 'swap,cipher=aes-xts-plain64'
ExecStop=/lib/systemd/systemd-cryptsetup detach 'crypt_swap'
ExecStartPost=/sbin/mkswap '/dev/mapper/crypt_swap'

What we think that the following is going on:

  • The systemd-cryptsetup attach call leads to systemd-udev creating the device /dev/mapper/crypt_setup with SYSTEMD_READY=0 as per /lib/udev/rules.d/99-systemd-rules and also starts setting up an inotify watch as per /lib/udev/rules.d/60-persistend-storage-dm.rules.
  • On slow systems the setup of this watch races with the mkswap call that is done next.
    If mkswap is called before the watch is set up by udev, it misses the fact that the swap partition is now ready, the device keeps being marked as SYSTEMD_READY=0.
  • As a result, the service dev-mapper-crypt_swap.device never becomes active and it blocks the boot because the job timeout is set to 0 for the service by a systemd override (in case the swap partition is considered "fail").

Below are the log files of the interaction of mkswap and systemd-udev for the cases where it goes right and wrong.

Correct

Sep 24 16:58:42 hostname systemd-cryptsetup[508]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev
...
Sep 24 16:58:46 hostname systemd-udevd[489]: handling device node '/dev/dm-1', devnum=b253:1, mode=0660, uid=0, gid=6
...
Sep 24 16:58:47 hostname systemd-udevd[485]: adding watch on '/dev/dm-1'
...
Sep 24 16:58:47 hostname mkswap[591]: Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
Sep 24 16:58:47 hostname mkswap[591]: no label, UUID=4a24083d-627c-416c-bbb0-aba431b9bfa5
...
Sep 24 16:58:47 hostname systemd-udevd[480]: inotify event: 8 for /dev/dm-1
...
Sep 24 16:58:47 hostname systemd-udevd[480]: device /dev/dm-1 closed, synthesising 'change'

Wrong

Sep 24 16:50:23 hostname systemd-cryptsetup[513]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev
...
Sep 24 16:50:27 hostname systemd-udevd[499]: handling device node '/dev/dm-1', devnum=b253:1, mode=0660, uid=0, gid=6
...
Sep 24 16:50:28 hostname mkswap[599]: Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
Sep 24 16:50:28 hostname mkswap[599]: no label, UUID=0a0ab441-8efe-44f3-92e4-2b8cb7a165c4
...
Sep 24 16:50:28 hostname systemd-udevd[498]: adding watch on '/dev/dm-1'

Considerations

While systemd often relies on proper notifications for other subsystems, such as LVM etc., in this case the setup is relatively bare and systemd itself calls the bare tool mkswap and relies on udev to mark the device as ready. We currently don't know how to solve this or get around the issue.

@arvidjaar

This comment has been minimized.

Copy link
Contributor

@arvidjaar arvidjaar commented Sep 30, 2018

creating the device /dev/mapper/crypt_setup with SYSTEMD_READY=0 as per /lib/udev/rules.d/99-systemd-rules

Wow! This is entirely bogus. There is no reason to prohibit use of device without recognizable format.

Actually #10154 is related. If systemd had ability to run commands to bring up device, this workaround would not be needed. It would just wait for mkswap to complete and proceed with any dependent services.

We currently don't know how to solve this or get around the issue.

Add udevadm trigger as one more ExecStartPost, after mkswap. This should reset SYSTEMD_READY.

@paulvt

This comment has been minimized.

Copy link
Author

@paulvt paulvt commented Oct 1, 2018

Indeed, I had already added the following override file systemd-cryptsetup@crypt_swap.service.d/90-trigger-udev.conf:

# Run udevadm trigger after the mkswap call in the original generated
# service

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

You seem to imply that the udev rules should not set SYSTEMD_READY=0 for a new crypt (opened) device, so changing that rule could be another workaround too?

@arvidjaar

This comment has been minimized.

Copy link
Contributor

@arvidjaar arvidjaar commented Oct 1, 2018

You seem to imply that the udev rules should not set SYSTEMD_READY=0 for a new crypt (opened) device, so changing that rule could be another workaround too?

Removing it now will break swap and tmp processing by starting followup units too early. This requires some non-trivial redesign of the whole mess.

@mmstick

This comment has been minimized.

Copy link

@mmstick mmstick commented Jul 23, 2019

Indeed, I had already added the following override file systemd-cryptsetup@crypt_swap.service.d/90-trigger-udev.conf:

# Run udevadm trigger after the mkswap call in the original generated
# service

[Service]
ExecStartPost=/sbin/udevadm trigger /dev/mapper/%i

That seems to do the job. This is an issue experienced heavily on Pop!_OS, because we encrypt swap partitions by default. Most hardware is unaffected, but there are a few models that hit this error regularly. After applying this, the issue stopped entirely on the Galago Pro 2.

Now I need to find a way to apply this automatically for all installations.

mmstick added a commit to mmstick/systemd that referenced this issue Jul 23, 2019
On some systems, encrypted swap partitions occasionally fail to mount at boot, because systemd does not receive the triggers that it is waiting to hear. By running `udevadm trigger /dev/mapper/$cryptswap` immediately after creation, this issue is resolved. Frequency of occurrence varies from system to system, so some systems will never experience the issue, others will occasionally experience it, and then there's the worst case scenario that experiences it almost every boot.

- Pop!_OS issue report: pop-os/pop#316
- Solution based on systemd#10179 (comment)
- Closes systemd#10179
@DemiMarie

This comment has been minimized.

Copy link

@DemiMarie DemiMarie commented Aug 20, 2019

I strongly suspect that the real bug is in systemd-udevd: it emits an add event before establishing an inotify watch. As a result, it races against mkswap.

@ndusart

This comment has been minimized.

Copy link

@ndusart ndusart commented Oct 30, 2019

Is there any plan to fix this ?

I'm experiencing this on Archlinux too. Most of the time, systemd will hang waiting for the swap partition while it is effectively created.

@grzegorzk

This comment has been minimized.

Copy link

@grzegorzk grzegorzk commented Jan 13, 2020

I'm experiencing this issue too. Fresh Archlinux installs, boot hangs while waiting for crypt-swap device. I experience this issue 100% boots on 1CPU/1Gb RAM Linode.

On fresh install with systemd 244 I configured crypttab and fstab swap entries with noauto

Starting auto-generated swap unit manually works 100% of times.

systemd-networkd will also fail if cryp-swap is enabled. Without crypt-swap systemd-networkd will not fail.

Hope this helps!

Applying the workaround with udevadm trigger works. This seems to be the best solution for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
7 participants
You can’t perform that action at this time.