ceph-disk: resolve race conditions #12136
Conversation
A ceph udev action may be triggered before the local file systems are mounted because there is no ordering in udev. The ceph udev action delegates asynchronously to systemd via ceph-disk@.service which will fail if (for instance) the LVM partition required to mount /var/lib/ceph is not available yet. The systemd unit will retry a few times but will eventually fail permanently. The sysadmin can systemctl reset-fail at a later time and it will succeed. Add a dependency to ceph-disk@.service so that it waits until the local file systems are mounted: After=local-fs.target Since local-fs.target depends on lvm, it will wait until the lvm partition (as well as any dm devices) is ready and mounted before attempting to activate the OSD. It may still fail because the corresponding journal/data partition is not ready yet (which is expected) but it will no longer fail because the lvm/filesystems/dm are not ready. Fixes: http://tracker.ceph.com/issues/17889 Signed-off-by: Loic Dachary <loic@dachary.org>
The udev rules that set the owner/group of the OSD devices are racing with 50-udev-default.rules and depending on which udev event fires last, ownership may not be as expected. Since ceph-disk trigger --sync runs as root, always happens after dm/lvm/filesystem units are complete and before activation, it is a good time to set the ownership of the device. It does not eliminate all races: a script running after systemd local-fs.target and firing a udev event may create a situation where the permissions of the device are temporarily reverted while the activation is running. Fixes: http://tracker.ceph.com/issues/17813 Signed-off-by: Loic Dachary <loic@dachary.org>
|
jenkins test this please (http://tracker.ceph.com/issues/17991) |
ghost
changed the title
teuthology-suite -k distro --verbose --suite ceph-disk --suite-branch master --ceph wip-17889-systemd-order --machine-type vps --priority 101 machine_types/vps.yaml |
| @@ -1,5 +1,7 @@ | |||
| [Unit] | |||
| Description=Ceph disk activation: %f | |||
| After=local-fs.target | |||
| Wants=local-fs.target | |||
There was a problem hiding this comment.
Minor nitpick, but shouldn't this be Requires=?
You want this unit to fail if the local fs can't be mounted (although, if that happens, you have bigger problems).
There was a problem hiding this comment.
It would make sense to me also. I chose to go with the same kind of dependency as the other .service and .target defined by Ceph. As you say: if that is not present...we're in trouble big time ;-)
|
@tchaikov would you have time to take a look ? |
|
The first patch might actually have no effect, since this is the default behaviour.
Not that it can do any harm to set this explicitly, but it might not be the fix for the race condition we're looking for. |
|
@rubenk that would be really surprising because the observed behavior can only be explained by ceph-disk trying to activate an OSD before a local file system mounted on a LVM is available. Thanks a lot for the link, I'll explore that further. |
|
Indeed, it surprised me as well, but I just checked (note this is without lvm and without this PR): |
|
@rubenk thanks, that's very helpful :-) I'll resume investigations on monday since it seems the problem is still there. |
|
I reopened http://tracker.ceph.com/issues/17813 |
http://tracker.ceph.com/issues/17813
http://tracker.ceph.com/issues/17889