Devices transition plugged -> dead -> plugged is still not completely fixed #12953

fbuihuu · 2019-07-04T08:17:10Z

This happened on Tumbleweed which ships systemd v242.

For the complete debug logs, please consult https://bugzilla.suse.com/show_bug.cgi?id=1137373.

Except from the logs:

[ 25.202778] fyrfos systemd[1]: Switching root.
[ 36.627399] fyrfos systemd[1]: Starting udev Coldplug all Devices...
[ 36.714237] fyrfos systemd[1]: Started udev Coldplug all Devices.
[ 48.182070] fyrfos systemd[1]: Reloading.
[ 48.433421] fyrfos systemd[1]: Invoking unit coldplug() handlers…
[ 48.457773] fyrfos systemd[1]: dev-sdb5.device: Changed dead -> plugged
[ 48.466512] fyrfos systemd[1]: Invoking unit catchup() handlers…
[ 48.471053] fyrfos systemd[1]: dev-sdb5.device: Changed plugged -> dead
[ 48.741803] fyrfos systemd[1]: dev-sdb5.device: Changed dead -> plugged
[ 49.005134] fyrfos systemd[1]: local.mount: local.mount lost dependency BindsTo=dev-sdb5.device

This was already reported in #11997 but the fix was incorrect, see #12013 (comment)

fbuihuu · 2019-07-04T08:18:04Z

I think the regression is severe enough to deserve the v243 milestone.

yuwata · 2019-07-04T08:18:35Z

I've reopend the previous issue page...

fbuihuu · 2019-07-04T08:21:14Z

Damn I missed that and strangely enough I haven't received any notification about your comment in the other bug although I'm subscribed to it.

Maybe we should continue the discussion here ?

yuwata · 2019-07-04T08:28:46Z

OK, old one is closed again.

fbuihuu · 2019-07-05T16:12:51Z

@yuwata I started working on a possible fix for this, please have a look at: https://github.com/fbuihuu/systemd/commits/fix-inconsistent-device-state-transition

Rather than assuming that the device enumeration can be trusted based on the manager state (reloading vs reexecuting), we now simply look at the udev DB itself - ie if no single devices couldn't have been enumerated then we assume that the DB is not yet initialized [1] and thereby don't trust the enumeration phase. In this case we restore the serialized state and rely on events retriggering to update the view on device PID1 had before it reexecuted. [1] Maybe there's a better way to detect this state. Fixes: systemd#12953

yuwata · 2019-07-10T07:37:10Z

@fbuihuu Do you have a reproducer of this issue?

Rather than assuming that the device enumeration can be trusted based on the manager state (reloading vs reexecuting), we now simply look at the udev DB itself - ie if no single devices couldn't have been enumerated then we assume that the DB is not yet initialized [1] and thereby don't trust the enumeration phase. In this case we restore the serialized state and rely on events retriggering to update the view on device PID1 had before it reexecuted. [1] Maybe there's a better way to detect this state. Fixes: systemd#12953

fbuihuu · 2019-07-10T08:51:36Z

@yuwata no sorry. I tried to run "systemctl daemon-reload" right before systemd-udevd.service is started but it led to a broken system and I haven't found some time to investigate why.

Rather than assuming that the device enumeration can be trusted based on the manager state (reloading vs reexecuting), we now simply look at the udev DB itself - ie if no single devices couldn't have been enumerated then we assume that the DB is not yet initialized [1] and thereby don't trust the enumeration phase. In this case we restore the serialized state and rely on events retriggering to update the view on device PID1 had before it reexecuted. [1] Maybe there's a better way to detect this state. Fixes: systemd#12953

poettering · 2019-07-23T14:08:30Z

since this isn't clear year, let's move to v244

fbuihuu · 2019-07-23T15:59:35Z

See #13002 for more details.

fbuihuu · 2019-10-28T16:30:15Z

@yuwata, it seems that commit c6e892b also introduced a regression.

IIRC the commit removed the "plugged -> dead -> plugged" transition after switching root and devices in "plugged" state in initramfs stay in "plugged" state after switching root.

But this prevents device dependencies which are only defined in the root filesystem (i.e. not included in initramfs) to be pulled in and started.

fbuihuu · 2019-10-28T16:32:38Z

It would nice if all of these (new) issues could be fixed for good.

fbuihuu · 2019-10-31T07:47:57Z

@keszybz, IMHO we shouldn't postpone this (again).

fbuihuu · 2019-12-05T11:32:21Z

Another user affected by this issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1155170

fbuihuu · 2019-12-05T11:32:51Z

It would be great if such bug could be fixed now.

While the manager is reloaded, the device_monitor socket is kept open; outstanding uevents will be received sooner or later. Thus if a device isn't found during udev enumeration / coldplug, we shouldn't blindly assume it's gone. Possibly the reload operation took place before udev started, or had the time to re-probe previously discovered devices (this matters mainly during boot / coldplug). So if the device actually disappeared, leave it to the outstanding uevent to make this happen in systemd. Fixes systemd#12953.

Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

See the comments in the code. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215. Co-authored-by: Martin Wilck <mwilck@suse.com>

The issue systemd#12953 is caused by the following: On switching root, - deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT, - deserialized_state == DEVICE_PLUGGED, - enumerated_found == DEVICE_FOUND_MOUNT, On switching root, most devices are not found by the enumeration process. Hence, the device state is set to plugged by device_coldplug(), and then changed to the dead state in device_catchup(). So the corresponding mount point is unmounted. Later when the device is processed by udevd, it will be changed to plugged state again. The issue systemd#23208 is caused by the fact that generated udev database in initramfs and the main system are often different. So, the two issues have the same root; we should not honor DEVICE_FOUND_UDEV bit in the deserialized_found on switching root. This partially reverts c6e892b. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215. Co-authored-by: Martin Wilck <mwilck@suse.com>

This is a slightly different approach than the one taken by commit 75d7b59 to fix issue systemd#12953 and systemd#23208. This patch forces PID1 to forget all devices (except those with the "db_persist" option see below) that were known by PID1 before switching root by pretending that the devices were in DEAD state before being serialized. Hence no artificial "plugged->dead" state transitions happen when PID1 is reexecuting from a switch root followed by "dead->plugged" state transitions when all devices are coldplugged with the new set of udev rule from the host. As mentioned previously, devices with the "db_persistent" option are exceptions of the previously described mechanism. Since these devices remain in the udev DB even after the DB has been cleared, they still continue to be deserialized in plugged state and remain in this state hence following the description of the option. This should fix the regression introduced by 75d7b59. Fixes: systemd#23429 Replaces: systemd#23218

The issue systemd#12953 is caused by the following: On switching root, - deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT, - deserialized_state == DEVICE_PLUGGED, - enumerated_found == DEVICE_FOUND_MOUNT, On switching root, most devices are not found by the enumeration process. Hence, the device state is set to plugged by device_coldplug(), and then changed to the dead state in device_catchup(). So the corresponding mount point is unmounted. Later when the device is processed by udevd, it will be changed to plugged state again. The issue systemd#23208 is caused by the fact that generated udev database in initramfs and the main system are often different. So, the two issues have the same root; we should not honor DEVICE_FOUND_UDEV bit in the deserialized_found on switching root. This partially reverts c6e892b. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215. Co-authored-by: Martin Wilck <mwilck@suse.com> (cherry picked from commit 75d7b59)

The issue systemd#12953 is caused by the following: On switching root, - deserialized_found == DEVICE_FOUND_UDEV | DEVICE_FOUND_MOUNT, - deserialized_state == DEVICE_PLUGGED, - enumerated_found == DEVICE_FOUND_MOUNT, On switching root, most devices are not found by the enumeration process. Hence, the device state is set to plugged by device_coldplug(), and then changed to the dead state in device_catchup(). So the corresponding mount point is unmounted. Later when the device is processed by udevd, it will be changed to plugged state again. The issue systemd#23208 is caused by the fact that generated udev database in initramfs and the main system are often different. So, the two issues have the same root; we should not honor DEVICE_FOUND_UDEV bit in the deserialized_found on switching root. This partially reverts c6e892b. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215. Co-authored-by: Martin Wilck <mwilck@suse.com> (cherry picked from commit 75d7b59) [mwilck: fixes bsc#1137373] [mwilck: fixes bsc#1181658] [mwilck: fixes bsc#1194708] [mwilck: fixes bsc#1195157] [mwilck: fixes bsc#1197570]

fbuihuu added the regression ⚠️ A bug in something that used to work correctly and broke through some recent commit label Jul 4, 2019

fbuihuu added this to the v243 milestone Jul 4, 2019

yuwata added the pid1 label Jul 4, 2019

yuwata added the has-pr ✨ label Jul 15, 2019

poettering modified the milestones: v243, v244 Jul 23, 2019

yuwata mentioned this issue Sep 6, 2019

Device transitions plugged -> dead -> plugged during a transaction has unexpected results #13489

Open

keszybz modified the milestones: v244, v245 Oct 24, 2019

fbuihuu mentioned this issue Jan 31, 2020

unit: drop initrd-udevadm-cleanup-db.service #12148

Closed

keszybz removed this from the v245 milestone Feb 28, 2020

mwilck mentioned this issue Apr 28, 2022

Fixes for device activation while reloading and switching root #23215

Closed

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

b93f9fd

Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata mentioned this issue Apr 29, 2022

pid1: several fixlets for device handling #23218

Merged

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

b9b01c2

Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

1a6e46e

Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

64e16a3

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

acb65eb

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 29, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

d6121f5

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 30, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

317e779

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata added a commit to yuwata/systemd that referenced this issue Apr 30, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

1679f8d

See the comments in the code. This is based on the work by Martin Wilck. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215.

yuwata mentioned this issue Apr 30, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root #23234

Closed

yuwata added a commit to yuwata/systemd that referenced this issue Apr 30, 2022

core/device: ignore DEVICE_FOUND_UDEV on switching root

9ff3241

See the comments in the code. Fixes systemd#12953 and systemd#23208. Replaces systemd#23215. Co-authored-by: Martin Wilck <mwilck@suse.com>

yuwata modified the milestones: v252, v251 May 11, 2022

yuwata closed this as completed in #23218 May 12, 2022

mwilck mentioned this issue May 19, 2022

PR#23218 introduced a regression for devices with db_persist option #23429

Closed

PlaidCat mentioned this issue May 20, 2022

Sporadic Timed out waiting for device on boot #13775

Closed

yuwata mentioned this issue May 25, 2022

core/device: honor deserialized state and check device existence on switching-root #23507

Closed

NickCao mentioned this issue Nov 13, 2022

nixos/udev: enable initrd-udevadm-cleanup-db.service in systemd stage 1 NixOS/nixpkgs#185116

Merged

13 tasks

yuwata mentioned this issue Jan 30, 2023

On first boot systemd believes some successfully mounted block devices have been unplugged and unmounts them #26254

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Devices transition plugged -> dead -> plugged is still not completely fixed #12953

Devices transition plugged -> dead -> plugged is still not completely fixed #12953

fbuihuu commented Jul 4, 2019

fbuihuu commented Jul 4, 2019

yuwata commented Jul 4, 2019

fbuihuu commented Jul 4, 2019

yuwata commented Jul 4, 2019

fbuihuu commented Jul 5, 2019

yuwata commented Jul 10, 2019

fbuihuu commented Jul 10, 2019

poettering commented Jul 23, 2019

fbuihuu commented Jul 23, 2019

fbuihuu commented Oct 28, 2019 •

edited

fbuihuu commented Oct 28, 2019

fbuihuu commented Oct 31, 2019

fbuihuu commented Dec 5, 2019

fbuihuu commented Dec 5, 2019

Devices transition plugged -> dead -> plugged is still not completely fixed #12953

Devices transition plugged -> dead -> plugged is still not completely fixed #12953

Comments

fbuihuu commented Jul 4, 2019

fbuihuu commented Jul 4, 2019

yuwata commented Jul 4, 2019

fbuihuu commented Jul 4, 2019

yuwata commented Jul 4, 2019

fbuihuu commented Jul 5, 2019

yuwata commented Jul 10, 2019

fbuihuu commented Jul 10, 2019

poettering commented Jul 23, 2019

fbuihuu commented Jul 23, 2019

fbuihuu commented Oct 28, 2019 • edited

fbuihuu commented Oct 28, 2019

fbuihuu commented Oct 31, 2019

fbuihuu commented Dec 5, 2019

fbuihuu commented Dec 5, 2019

fbuihuu commented Oct 28, 2019 •

edited