PR#23218 introduced a regression for devices with db_persist option

[fbuihuu]

Devices with OPTIONS+="db_persist" are not supposed to create dead->plugged transition right after switching root, at least that wasn't the case before and the udev man page describes the option to actually prevent it from happening.

However pr #23218 changed this behavior. I don't know currently whether it will cause any real regression in practice but I think it deserves a discussion before releasing v251.

[mwilck]

I wouldn't call this a regression. No actual negative impact is known except the debug messages.

See also #23218 (comment)

[fbuihuu]

Hi Martin, it's pretty hard to say for sure that it won't have any negative impact but the facts are that this transition never happened before and the doc clearly describes the use of "db_persist" as way to make sure that the device unit state remains always plugged.

[yuwata]

@mwilck

#23218 (comment)

It's not clear to me whether it will cause issue in practice (maybe @mwilck knows ?) but this dead->plugged transition for devices with db_persist doen't look correct and wasn't present before. And udev man page seems to indicate that this transition should not occur.

Why would this even be mentioned in an the udev man page? It's genuine systemd functionality.

I have observed these transitions (I see them all the time actually because I work with multipath-tools, but I didn't bother about it), but definitely no negative impact.

I have a test case here where root FS (btrfs) is on top of LVM on top of MD RAID on top of multipath. Thus the multipath devices (which have db_persist set) are in use once root is mounted, but from the PoV of systemd, they have neither DEVICE_FOUND_MOUNT nor DEVICE_FOUND_SWAP set. Thus they start out "dead" after switching root, just like in @fbuihuu's example. I can see these dead→plugged transitions with loglevel debug, but everything works just fine during boot.

#23218 (comment)

@yuwata. recall this discussion we had previously? If we removed the lines

+                if (found == DEVICE_NOT_FOUND)
+                        state = DEVICE_DEAD; /* If nobody sees the device, downgrade more */

from 75d7b59, we'd still see transition messages but only "tentative→plugged", not the confusing "dead→plugged". I don't think this can be completely avoided. We apply the result from enumeration after the result from deserialization on purpose.

Yeah, the condition seems slightly offensive, I will fix it after dinner. But, the basic idea of the block is that setting DEVICE_TENTATIVE with DEVICE_NOT_FOUND is super strange.

[mwilck]

But, the basic idea of the block is that setting DEVICE_TENTATIVE with DEVICE_NOT_FOUND is super strange.

Not if you consider complex storage stacks. If a device has been "plugged" but not mounted (instead, member of an MD RAID, or an LVM PV or what not), and you mask out DEVICE_FOUND_UDEV, you get this combination. You can of course try to build a model of complex storage stacks in systemd, but I wouldn't recommend it, at least not short term.

DEVICE_TENTATIVE makes a lot of sense for me in this case. At the given point in time (before device_catchup(), systemd has no confirmation from udev whether the device exists, but it existed before, which makes "tentative" a reasonable guess.

[mwilck]

Hi Martin, it's pretty hard to say for sure that it won't have any negative impact but the facts are that this transition never happened before and the doc clearly describes the use of "db_persist" as way to make sure that the device unit state remains always plugged.

Correct me if I'm wrong: the issue in #12953 was that we observed plugged→dead transitions. These transitions are fatal, as are dead→plugged transitions based on "guessing" (rather than actual udev device detection). Both these types of wrong transitions are fixed by @yuwata's patch set.

What you've been observing is that db_persist devices undergo dead→plugged transitions repeatedly. The first transition is "real", while the others are just caused by the fact that the devices start out "dead" after re-execution unless they're mounted. I am quite positive that this is harmless, and I am not sure if it can be avoided. We would need to start out with DEVICE_PLUGGED, DEVICE_FOUND_UDEV if enumerated_state == DEVICE_PLUGGED, enumerated_found == DEVICE_FOUND_UDEV after device_enumerate(). While that would fix your issue, my gut feeling is that it might cause problems in other cases like the "systemd reloads while booting" case that we've discussed elsewhere.

[yuwata]

@mwilck I'd like to recall that it is dangerous to set DEVICE_TENTATIVE with DEVICE_NOT_FOUND. If we do so in device_coldplug(), then the device will not enter the dead state even if enumerated_found == DEVICE_NOT_FOUND, and it is stuck in the tentative state, and no corresponding unit will be stopped.

[yuwata]

If a device has persistent database, then the Device.enumerated_found can be 'reliable' in some sense, and if it has DEVICE_FOUND_UDEV flag, it is not necessary to downgrade the flag and state, as device_catchup() will fix the state soon later. If we adjust the found and state in device_coldplug(), it generates unnecessary state transition, as reported in this issue.

[yuwata]

I guess the unnecessary state transition reported in this issue triggers the following mount unit being stopped:

[Unit]
Requires=sys-devices-WithPersistentDatabase.device  <-- maybe BoundTo=??
After=sys-devices-WithPersistentDatabase.device

[Mount]
What=/dev/BlockWithoutPersistentDatabase
Where=/mnt/foo

Here, important point is the devices in What= and Requires= are different.
If the block device corresponds to sys-devices-WithPersistentDatabase.device is not mounted, then its Device.enumerated_found == DEVICE_NOT_FOUND on switching root and it enters DEVICE_DEAD, hence even if the device unit corresponds to /dev/BlockWithoutPersistentDatabase is in the tentative state, the mount unit is stopped during the switching root.

[mwilck]

@mwilck I'd like to recall that it is dangerous to set DEVICE_TENTATIVE with DEVICE_NOT_FOUND. If we do so in device_coldplug(), then the device will not enter the dead state even if enumerated_found == DEVICE_NOT_FOUND, and it is stuck in the tentative state, and no corresponding unit will be stopped.

IIUC "tentative" corresponds to "activating", and thus will be subject to systemd's general timeout handling for unit activation. IOW if a device was actually "plugged" before switching root, but not found any more in the root FS, it would switch to "dead" state when the timeout expires. Am I missing something?

As noted before, the combination DEVICE_PLUGGED, DEVICE_FOUND_UDEV frequently occurs with multipath, MD, LVM etc. If we mask out DEVICE_FOUND_UDEV after switching root, we end up with found = DEVICE_NOT_FOUND. Resetting state from DEVICE_PLUGGED to DEVICE_DEAD in this case sounds counter-intuitive to me; calling device_set_state(DEVICE_DEAD) would be outright dangerous because it would stop dependent units.

Here, important point is the devices in What= and Requires= are different. If the block device corresponds to sys-devices-WithPersistentDatabase.device is not mounted, then its Device.enumerated_found == DEVICE_NOT_FOUND on switching root and it enters DEVICE_DEAD

But nothing bad would happen with DEVICE_TENTATIVE.

[keszybz]

I don't feel like I understand all the details here… but based on @mwilck's and @yuwata's comments, it seems that the issue does not cause a regression (in the sense of an actual user-visible problem). So I think we should release v251 in the current state.

[mwilck]

I've found additional evidence that this is not a regression. See #23437 (comment).

[keszybz]

I suspect https://bugzilla.redhat.com/show_bug.cgi?id=2088788, https://bugzilla.redhat.com/show_bug.cgi?id=2087225, and coreos/fedora-coreos-tracker#1200 could be related. I'll ask the reporters to test with the patch reverted. I'm doing a scratch build in koji right now.

[tanriol]

My system failed to boot with systemd 251, and reverting #23218 fixed that.
Gentoo Linux, with initrd (dracut), with LUKS. Configured manually, so the configuration can be unusual in some aspects.
Here's a slightly modified diff of the corresponding part of boot logs (sorry, normal verbosity, not debug)

diff

(no interesting changes before switch root)
 systemd 251 running in system mode (+PAM -AUDIT -SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
 Detected architecture x86-64.
 initrd-switch-root.service: Deactivated successfully.
 Stopped initrd-switch-root.service.
 systemd-journald.service: Scheduled restart job, restart counter is at 1.
 Created slice system-getty.slice.
 Created slice system-modprobe.slice.
 Created slice system-systemd\x2dfsck.slice.
 Created slice user.slice.
 Set up automount proc-sys-fs-binfmt_misc.automount.
 systemd-ask-password-console.path was skipped because of a failed condition check (ConditionPathExists=!/run/plymouth/pid).
 Started systemd-ask-password-wall.path.
-Reached target blockdev@dev-mapper-px_root.target.
-Reached target blockdev@dev-mapper-px_swap.target.
 Reached target getty.target.
 Stopped target initrd-switch-root.target.
 Stopped target initrd-fs.target.
 Stopped target initrd-root-fs.target.
 Reached target integritysetup.target.
 Reached target remote-cryptsetup.target.
 Reached target remote-fs.target.
 Reached target slices.target.
 Reached target veritysetup.target.
 Listening on systemd-coredump.socket.
 Listening on systemd-initctl.socket.
 Listening on systemd-networkd.socket.
 Listening on systemd-udevd-control.socket.
-Listening on systemd-udevd-kernel.socket.
-Activating swap dev-mapper-px_swap.swap...
 dev-hugepages.mount was skipped because of a failed condition check (ConditionPathExists=/sys/kernel/mm/hugepages).
 Mounting dev-mqueue.mount...
 Mounting sys-kernel-debug.mount...
 Mounting sys-kernel-tracing.mount...
 Starting kmod-static-nodes.service...
 Starting modprobe@configfs.service...
 Starting modprobe@drm.service...
 Starting modprobe@fuse.service...
 plymouth-switch-root.service: Deactivated successfully.
 Stopped plymouth-switch-root.service.
-Adding 67106812k swap on /dev/mapper/px_swap.  Priority:-2 extents:1 across:67106812k SS
-Starting systemd-cryptsetup@px_home.service...
-Starting systemd-cryptsetup@storage.service...
-Starting systemd-cryptsetup@storage\x2dvm.service...
-Starting systemd-cryptsetup@storage\x2dxl.service...
+Stopping systemd-cryptsetup@px_root.service...
+Stopping systemd-cryptsetup@px_swap.service...
 Stopped systemd-journald.service.
 Starting systemd-journald.service...
 Starting systemd-modules-load.service...
 fuse: init (API version 7.36)
 Starting systemd-remount-fs.service...
 systemd-repart.service was skipped because all trigger condition checks failed.
-Starting systemd-udev-trigger.service...
-Activated swap dev-mapper-px_swap.swap.
 Mounted dev-mqueue.mount.
 Mounted sys-kernel-debug.mount.
 Mounted sys-kernel-tracing.mount.
 Finished kmod-static-nodes.service.
 modprobe@configfs.service: Deactivated successfully.
 Finished modprobe@configfs.service.
 modprobe@drm.service: Deactivated successfully.
 Finished modprobe@drm.service.
-EXT4-fs (dm-0): re-mounted. Quota mode: none.
 modprobe@fuse.service: Deactivated successfully.
 Finished modprobe@fuse.service.
+systemd-cryptsetup@px_root.service: Control process exited, code=exited, status=1/FAILURE
+systemd-cryptsetup@px_root.service: Failed with result 'exit-code'.
+Stopped systemd-cryptsetup@px_root.service.
 Finished systemd-modules-load.service.
-Finished systemd-remount-fs.service.
-Reached target swap.target.
-Journal started
-Runtime Journal (/run/log/journal/40a7ddbe283140c18111fd16b04c18f6) is 8.0M, max 638.6M, 630.6M free.
-Queued start job for default target graphical.target.
-systemd-journald.service: Deactivated successfully.
-Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-partlabel/storage.
-Set cipher aes, mode xts-plain64, key size 256 bits for device /dev/disk/by-partlabel/px_home.
-Failed to find module 'ipmi-devintf'
-Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-partlabel/storage-vm.
-Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-partlabel/storage-xl.
+Reached target blockdev@dev-mapper-px_root.target.
 Mounting sys-fs-fuse-connections.mount...
 Mounting sys-kernel-config.mount...
+Starting systemd-sysctl.service...
+Stopped target blockdev@dev-mapper-px_root.target.
+Mounted sys-fs-fuse-connections.mount.
+Mounted sys-kernel-config.mount.
+EXT4-fs (dm-0): re-mounted. Quota mode: none.
+Finished systemd-remount-fs.service.
 systemd-firstboot.service was skipped because of a failed condition check (ConditionFirstBoot=yes).
 systemd-hwdb-update.service was skipped because of a failed condition check (ConditionNeedsUpdate=/etc).
 systemd-pstore.service was skipped because of a failed condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
 Starting systemd-random-seed.service...
-Starting systemd-sysctl.service...
 systemd-sysusers.service was skipped because of a failed condition check (ConditionNeedsUpdate=/etc).
 Starting systemd-tmpfiles-setup-dev.service...
+Finished systemd-sysctl.service.
+Journal started
+Runtime Journal (/run/log/journal/40a7ddbe283140c18111fd16b04c18f6) is 8.0M, max 638.6M, 630.6M free.
+Queued start job for default target graphical.target.
+Unnecessary job was removed for dev-disk-by\x2duuid-1c281d44\x2d991d\x2d4121\x2da855\x2d0f01d9c5512e.device.
+Unnecessary job was removed for dev-disk-by\x2duuid-c1adc4ce\x2dd27e\x2d4d9f\x2d859e\x2dcf58d725f773.device.
+systemd-journald.service: Deactivated successfully.
+Device px_root is still in use.
 Started systemd-journald.service.
-Mounted sys-fs-fuse-connections.mount.
-Mounted sys-kernel-config.mount.
+Failed to deactivate: Device or resource busy
(from this point the logs diverge almost completely)

[mwilck]

Sigh… so it seems we have an issue with LUKS, or dm-crypt in general. That matches also some of the bugs mentioned in #23429 (comment).

@tanriol, can you please describe your block device stack, and provide journalctl -o short-monotonic -b taken with systemd.log_level=debug? Could you test if the changes in #23437 fix our issue?

[mwilck]

dm-crypt devices have db_persist set, AFAIK. So the fix from #23437 might actually help. And indeed, this is a very good reason to use DEVICE_READY for such devices rather than DEVICE_TENTATIVE.

I suppose this is a fallout of the fact that systemd recognizes devices used by mounts or swaps, but no other, more complex block device dependencies.

[tanriol]

@tanriol, can you please describe your block device stack

Nothing really fancy - multiple separate disks, each has GPT, LUKS1, filesystems. fstab mentions devices by their DM locations like /dev/mapper/px_root, crypttab mentions most devices by PARTLABEL= except px_root and px_swap that use UUID=. Dracut's config only explicitly mentions /dev/mapper/px_swap, otherwise left for dracut to deduct.

and provide journalctl -o short-monotonic -b taken with systemd.log_level=debug?

For a failing boot? In a couple days, I think, when I have time to test that.

Could you test if the changes in #23437 fix our issue?

Okay, will check when I have time.

[mwilck]

I've pushed another patch to #23489 which fixes the LUKS issue in my test setup.

Analysis:

dm-crypt device units generated by systemd-cryptsetup-generator
have BindsTo= dependencies on their backend devices. The dm-crypt
devices have the db_persist flag set, and thus survive the udev db
cleanup while switching root. But backend devices usually don't survive.
These devices are neither mounted nor used for swap, thus they will
seen as DEVICE_NOT_FOUND after switching root.

The BindsTo dependency will cause systemd to schedule a stop
job for the dm-crypt device, breaking boot:

    [   68.929457] krypton systemd[1]: systemd-cryptsetup@cr_root.service: Unit is stopped because bound to inactive unit dev-disk-by\x2duuid-3bf91f73\x2d1ee8\x2d4cfc\x2d9048\x2d93ba349b786d.device.
    [   68.945660] krypton systemd[1]: systemd-cryptsetup@cr_root.service: Trying to enqueue job systemd-cryptsetup@cr_root.service/stop/replace
    [   69.473459] krypton systemd[1]: systemd-cryptsetup@cr_root.service: Installed new job systemd-cryptsetup@cr_root.service/stop as 343

By not setting the state of these devices to DEVICE_DEAD but DEVICE_TENTATIVE, the issue can be avoided. @yuwata, you certainly recall our past discussion about this hunk.

[mwilck]

It'd be grately appreciated if @tanriol and the other users affected by the regression could test with the patches from #23489. Again I'm unsure how to provide binaries to them for testing.