Skip to content

/ systemd

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mkfs.btrfs on encrypted devices causes systemd-cryptsetup restart loop #5955

Open
arvidjaar opened this issue May 13, 2017 · 3 comments
Open

mkfs.btrfs on encrypted devices causes systemd-cryptsetup restart loop #5955

arvidjaar opened this issue May 13, 2017 · 3 comments
Labels
pid1

Comments

@arvidjaar
Copy link
Contributor

@arvidjaar arvidjaar commented May 13, 2017

systemd 232 (openSUSE Tumbleweed).

Initial state - existing btrfs on three encrypted devices:

10:~ # cat /etc/crypttab 
storage1 UUID=ad138cdc-897d-435c-b4a1-c24ebec1cf6e /key
storage2 UUID=1dce34c6-6615-4e70-8230-92abf9c7675c /key
storage3 UUID=9badc783-b43c-415c-b4e1-a4a0a95e2dd0 /key
10:~ # ll /dev/mapper/
total 0
crw------- 1 root root 10, 236 May 13 21:28 control
lrwxrwxrwx 1 root root       7 May 13 21:28 storage1 -> ../dm-2
lrwxrwxrwx 1 root root       7 May 13 21:28 storage2 -> ../dm-1
lrwxrwxrwx 1 root root       7 May 13 21:28 storage3 -> ../dm-0
10:~ # systemctl status -l --no-pager systemd-cryptsetup@storage\*
● systemd-cryptsetup@storage1.service - Cryptography Setup for storage1
   Loaded: loaded (/etc/crypttab; generated; vendor preset: disabled)
   Active: active (exited) since Sat 2017-05-13 21:28:31 MSK; 1min 35s ago
     Docs: man:crypttab(5)
           man:systemd-cryptsetup-generator(8)
           man:systemd-cryptsetup@.service(8)
  Process: 883 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach storage1 /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e /key  (code=exited, status=0/SUCCESS)
 Main PID: 883 (code=exited, status=0/SUCCESS)

May 13 21:28:24 localhost systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:28:24 localhost systemd-cryptsetup[883]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.
May 13 21:28:31 localhost systemd[1]: Started Cryptography Setup for storage1.

● systemd-cryptsetup@storage2.service - Cryptography Setup for storage2
   Loaded: loaded (/etc/crypttab; generated; vendor preset: disabled)
   Active: active (exited) since Sat 2017-05-13 21:28:31 MSK; 1min 35s ago
     Docs: man:crypttab(5)
           man:systemd-cryptsetup-generator(8)
           man:systemd-cryptsetup@.service(8)
  Process: 933 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach storage2 /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c /key  (code=exited, status=0/SUCCESS)
 Main PID: 933 (code=exited, status=0/SUCCESS)

May 13 21:28:24 localhost systemd[1]: Starting Cryptography Setup for storage2...
May 13 21:28:24 localhost systemd-cryptsetup[933]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c.
May 13 21:28:31 localhost systemd[1]: Started Cryptography Setup for storage2.

● systemd-cryptsetup@storage3.service - Cryptography Setup for storage3
   Loaded: loaded (/etc/crypttab; generated; vendor preset: disabled)
   Active: active (exited) since Sat 2017-05-13 21:28:31 MSK; 1min 35s ago
     Docs: man:crypttab(5)
           man:systemd-cryptsetup-generator(8)
           man:systemd-cryptsetup@.service(8)
  Process: 879 ExecStart=/usr/lib/systemd/systemd-cryptsetup attach storage3 /dev/disk/by-uuid/9badc783-b43c-415c-b4e1-a4a0a95e2dd0 /key  (code=exited, status=0/SUCCESS)
 Main PID: 879 (code=exited, status=0/SUCCESS)

May 13 21:28:24 localhost systemd[1]: Starting Cryptography Setup for storage3...
May 13 21:28:24 localhost systemd-cryptsetup[879]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/9badc783-b43c-415c-b4e1-a4a0a95e2dd0.
May 13 21:28:31 localhost systemd[1]: Started Cryptography Setup for storage3.
10:~ # systemctl list-jobs
JOB UNIT                       TYPE  STATE  
168 dev-mapper-storage2.device start running
173 dev-mapper-storage3.device start running

2 jobs listed.

Of course two devices still "do not exist" because we helpfully mark them as "not ready".

10:~ # udevadm info --export-db | grep -E 'P: .*dm-|BTRFS_READY|SYSTEMD_READY'
E: ID_BTRFS_READY=1
P: /devices/virtual/block/dm-0
E: ID_BTRFS_READY=0
E: SYSTEMD_READY=0
P: /devices/virtual/block/dm-1
E: ID_BTRFS_READY=0
E: SYSTEMD_READY=0
P: /devices/virtual/block/dm-2
E: ID_BTRFS_READY=1

Now create new filesystem on the same device.

10:~ # mkfs.btrfs -f /dev/mapper/storage[1-3]
btrfs-progs v4.10.2+20170406
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               8df01d71-73fb-4619-b77c-7a9bbceda805
Node size:          16384
Sector size:        4096
Filesystem size:    59.99GiB
Block group profiles:
  Data:             RAID0             3.00GiB
  Metadata:         RAID1             1.00GiB
  System:           RAID1             8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  3
Devices:
   ID        SIZE  PATH
    1    20.00GiB  /dev/mapper/storage1
    2    20.00GiB  /dev/mapper/storage2
    3    20.00GiB  /dev/mapper/storage3

10:~ # ll /dev/mapper
total 0
crw------- 1 root root 10, 236 May 13 21:28 control
lrwxrwxrwx 1 root root       7 May 13 21:33 storage3 -> ../dm-0

Oops. Two of three devices are gone. systemd goes amok stopping and starting cryptsetup services until it finally manages to somehow stabilize.

May 13 21:33:19 10 systemd[1]: Found device /dev/mapper/storage3.
May 13 21:33:19 10 systemd[1]: Found device /dev/mapper/storage2.
May 13 21:33:19 10 systemd[1]: Startup finished in 1.410s (kernel) + 2.619s (initrd) + 4min 54.780s (userspace) = 4min 58.810s.
May 13 21:33:19 10 kernel: BTRFS: device fsid 8df01d71-73fb-4619-b77c-7a9bbceda805 devid 1 transid 5 /dev/dm-2
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-uuid/75da2441-d676-45ff-bca4-27edd7dddb0a.
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/raid-storage1.
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-ad138cdc897d435cb4a1c24ebec1cf6e-storage1.
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/dm-name-storage1.
May 13 21:33:19 10 systemd[1]: Stopped /dev/dm-2.
May 13 21:33:19 10 systemd[1]: Stopped /sys/devices/virtual/block/dm-2.
May 13 21:33:19 10 systemd[1]: Stopped target Encrypted Volumes.
May 13 21:33:19 10 systemd[1]: Stopping Cryptography Setup for storage1...
May 13 21:33:19 10 kernel: BTRFS: device fsid 8df01d71-73fb-4619-b77c-7a9bbceda805 devid 2 transid 5 /dev/dm-1
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/raid-storage2.
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-1dce34c666154e70823092abf9c7675c-storage2.
May 13 21:33:19 10 systemd[1]: Stopped /dev/disk/by-id/dm-name-storage2.
May 13 21:33:19 10 systemd[1]: Stopped /dev/dm-1.
May 13 21:33:19 10 systemd[1]: Stopped /sys/devices/virtual/block/dm-1.
May 13 21:33:19 10 systemd[1]: Stopping Cryptography Setup for storage2...
May 13 21:33:19 10 systemd[1]: Stopped Cryptography Setup for storage2.
May 13 21:33:19 10 kernel: BTRFS: device fsid 8df01d71-73fb-4619-b77c-7a9bbceda805 devid 3 transid 5 /dev/mapper/storage3
May 13 21:33:19 10 systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:33:19 10 systemd-cryptsetup[3079]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.
May 13 21:33:22 10 systemd[1]: Started Cryptography Setup for storage1.
May 13 21:33:22 10 systemd[1]: systemd-cryptsetup@storage1.service: Unit is bound to inactive unit dev-mapper-storage1.device. Stopping, too.
May 13 21:33:22 10 systemd[1]: Stopping Cryptography Setup for storage1...
May 13 21:33:22 10 systemd[1]: Found device /dev/mapper/storage2.
May 13 21:33:22 10 systemd[1]: Starting Cryptography Setup for storage2...
May 13 21:33:22 10 systemd-cryptsetup[3103]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c.
May 13 21:33:22 10 systemd[1]: systemd-cryptsetup@storage2.service: Main process exited, code=killed, status=15/TERM
May 13 21:33:22 10 systemd[1]: Stopped Cryptography Setup for storage2.
May 13 21:33:22 10 systemd[1]: systemd-cryptsetup@storage2.service: Unit entered failed state.
May 13 21:33:22 10 systemd[1]: systemd-cryptsetup@storage2.service: Failed with result 'signal'.
May 13 21:33:22 10 systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:33:22 10 systemd-cryptsetup[3108]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.
May 13 21:33:24 10 systemd[1]: Started Cryptography Setup for storage1.
May 13 21:33:24 10 systemd[1]: systemd-cryptsetup@storage1.service: Unit is bound to inactive unit dev-mapper-storage1.device. Stopping, too.
May 13 21:33:24 10 systemd[1]: Stopping Cryptography Setup for storage1...
May 13 21:33:24 10 systemd[1]: Found device /dev/mapper/storage2.
May 13 21:33:24 10 systemd[1]: Starting Cryptography Setup for storage2...
May 13 21:33:24 10 systemd-cryptsetup[3131]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c.
May 13 21:33:24 10 systemd[1]: systemd-cryptsetup@storage2.service: Main process exited, code=killed, status=15/TERM
May 13 21:33:24 10 systemd[1]: Stopped Cryptography Setup for storage2.
May 13 21:33:24 10 systemd[1]: systemd-cryptsetup@storage2.service: Unit entered failed state.
May 13 21:33:24 10 systemd[1]: systemd-cryptsetup@storage2.service: Failed with result 'signal'.
May 13 21:33:24 10 systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:33:24 10 systemd-cryptsetup[3136]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.
May 13 21:33:27 10 systemd[1]: Started Cryptography Setup for storage1.
May 13 21:33:27 10 systemd[1]: systemd-cryptsetup@storage1.service: Unit is bound to inactive unit dev-mapper-storage1.device. Stopping, too.
May 13 21:33:27 10 systemd[1]: Stopping Cryptography Setup for storage1...
May 13 21:33:27 10 systemd[1]: Found device /dev/mapper/storage2.
May 13 21:33:27 10 systemd[1]: Starting Cryptography Setup for storage2...
May 13 21:33:27 10 systemd-cryptsetup[3160]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c.
May 13 21:33:27 10 systemd[1]: systemd-cryptsetup@storage2.service: Main process exited, code=killed, status=15/TERM
May 13 21:33:27 10 systemd[1]: Stopped Cryptography Setup for storage2.
May 13 21:33:27 10 systemd[1]: systemd-cryptsetup@storage2.service: Unit entered failed state.
May 13 21:33:27 10 systemd[1]: systemd-cryptsetup@storage2.service: Failed with result 'signal'.
May 13 21:33:27 10 systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:33:27 10 systemd-cryptsetup[3165]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.
May 13 21:33:29 10 systemd[1]: Started Cryptography Setup for storage1.
May 13 21:33:29 10 systemd[1]: systemd-cryptsetup@storage1.service: Unit is bound to inactive unit dev-mapper-storage1.device. Stopping, too.
May 13 21:33:29 10 systemd[1]: Stopping Cryptography Setup for storage1...
May 13 21:33:29 10 systemd[1]: Found device /dev/mapper/storage2.
May 13 21:33:29 10 systemd[1]: Starting Cryptography Setup for storage2...
May 13 21:33:29 10 systemd-cryptsetup[3188]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/1dce34c6-6615-4e70-8230-92abf9c7675c.
May 13 21:33:29 10 systemd[1]: systemd-cryptsetup@storage2.service: Main process exited, code=killed, status=15/TERM
May 13 21:33:29 10 systemd[1]: Stopped Cryptography Setup for storage2.
May 13 21:33:29 10 systemd[1]: systemd-cryptsetup@storage2.service: Unit entered failed state.
May 13 21:33:29 10 systemd[1]: systemd-cryptsetup@storage2.service: Failed with result 'signal'.
May 13 21:33:29 10 systemd[1]: Starting Cryptography Setup for storage1...
May 13 21:33:29 10 systemd-cryptsetup[3193]: Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-uuid/ad138cdc-897d-435c-b4a1-c24ebec1cf6e.

10:~ # ll /dev/mapper
total 0
crw------- 1 root root 10, 236 May 13 21:28 control
lrwxrwxrwx 1 root root       7 May 13 21:34 storage1 -> ../dm-1
lrwxrwxrwx 1 root root       7 May 13 21:34 storage2 -> ../dm-2
lrwxrwxrwx 1 root root       7 May 13 21:33 storage3 -> ../dm-0

10:~ # udevadm info --export-db | grep -E 'P: .*dm-|BTRFS_READY|SYSTEMD_READY'
E: ID_BTRFS_READY=1
P: /devices/virtual/block/dm-0
E: ID_BTRFS_READY=1
P: /devices/virtual/block/dm-1
E: ID_BTRFS_READY=1
P: /devices/virtual/block/dm-2
E: ID_BTRFS_READY=1
@tomty89
Copy link
Contributor

@tomty89 tomty89 commented May 14, 2017 •
edited

https://github.com/systemd/systemd/blob/master/rules/99-systemd.rules.in#L20
https://github.com/systemd/systemd/blob/master/src/cryptsetup/cryptsetup-generator.c#L112

Note that the rule is not limited to add but also change action, which will be triggered when you mkfs/wipefs.

Since you were using mkfs.btrfs -f, it implies that there might be existing filesystem signatures on the mappers.

udev might be able to notice the "blank stage" of the mappers after mkfs.btrfs wiped the existing signatures before it had the new filesystem written, and hence mark SYSTEMD_READY to 0, which made systemd consider the mapper devices "unplugged", which in turn caused the systemd-cryptsetup services to be stopped because they bind to their mapper devices. (When they were stopped, the containers were closed and hence the "actual" mappers devices were gone.)
@ochilan
Copy link

@ochilan ochilan commented May 14, 2017

Intuitively I think the dm devices should simply be left alone once they have been opened. In any case, the raw devices should not vanish no matter their contents. That's the wrong direction to go in the depencency chain. In particular, having no FS on a dm should not mean it is unplugged. That should depend on the underlying device, not the user contents in the mapper.

Systemd really makes some complicated things simpler, but it makes some simple things so complicated (SCNR).
@tomty89
Copy link
Contributor

@tomty89 tomty89 commented May 14, 2017 •
edited

Well, let say you have the encrypted partition opened by crypttab/systemd-cryptsetup service, but then you cryptsetup close the mapper (insteading of stopping the service) later, the BindsTo= (at least the stop on fail part) make sense because it will cause the service to be stopped.

I am not sure about the point of the udev rule. At the very least, it should be limited to add action I suppose.

P.S. I've been having concern over BindsTo= recently, for the fact that it also implies a Requires=, especially when start job does not really make sense to devices...
@tomty89 tomty89 mentioned this issue May 24, 2017
Closed
@poettering poettering added the pid1 label Jul 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pid1
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
@arvidjaar @poettering @ochilan @tomty89
You can’t perform that action at this time.