systemd attempted to unmount filesystem after btrfs replace on LUKS

[intelfx]

Probably related to #14674, #14454.

systemd version the issue has been seen with

systemd 248 (248-5-arch)

Used distribution

Arch

Linux kernel version used (uname -a)

Linux anystation 5.11.15-arch1pf8-1 #2 SMP PREEMPT Thu, 01 Jan 1970 00:00:00 +0000 x86_64 GNU/Linux

CPU architecture issue was seen on

x86_64

Expected behaviour you didn't see
After btrfs replace, system should operate normally.

Unexpected behaviour you saw

After completion of btrfs replace, systemd attempted to unmount the filesystem in question from all mountpoints (filesystem in question is the root filesystem, but with several bind-mounts and subvolume mounts to other locations) and stop all dependent units:

Additionally, the .device unit corresponding to the new volume (replace target) got stuck in the activating (tentative) state.

Steps to reproduce the problem

1. Set up an Arch installation with btrfs root on LUKS on GPT (not sure if either of this is relevant, but including for completeness)
2. Create additional subvolumes, add them to /etc/fstab to be mounted at separate locations and mount them
3. Optionally, create and start services dependent on these mountpoints (Requires=, RequiresMountsFor=) for better demonstration purposes
4. Add another disk or partition into the system, create and open a LUKS container on it
5. Initiate a btrfs replace to the new LUKS volume and wait for it to finish

Additional program output to the terminal or log subsystem illustrating the issue

System log:

Apr 22 06:49:16 anystation kernel: BTRFS info (device dm-0): dev_replace from /dev/mapper/root (devid 2) to /dev/mapper/new finished
Apr 22 06:49:16 anystation systemd[1]: Stopped target Local File Systems.
Apr 22 06:49:16 anystation systemd-nspawn[3690]: stratofortress login: Trying to halt container. Send SIGTERM again to trigger immediate termination.
Apr 22 06:49:16 anystation systemd[1]: Unmounting /var/cache/pacman/custom...
Apr 22 06:49:16 anystation systemd[1]: Unmounting /var/cache/pacman/pkg...
Apr 22 06:49:16 anystation systemd[1]: Unmounting /var/lib/libvirt...
Apr 22 06:49:16 anystation systemd[1]: Stopping Virtual Machine and Container Registration Service...
Apr 22 06:49:16 anystation umount[146600]: umount: /var/lib/libvirt: target is busy.
Apr 22 06:49:16 anystation systemd[1]: Stopping Container stratofortress...
Apr 22 06:49:16 anystation systemd[1]: systemd-sysctl.service: Deactivated successfully.
Apr 22 06:49:16 anystation systemd[1]: Stopped Apply Kernel Variables.
Apr 22 06:49:16 anystation systemd[1260]: var-cache-pacman-custom.mount: Deactivated successfully.
Apr 22 06:49:16 anystation systemd[1]: var-cache-pacman-custom.mount: Deactivated successfully.
Apr 22 06:49:16 anystation systemd[1260]: var-cache-pacman-pkg.mount: Deactivated successfully.
Apr 22 06:49:16 anystation systemd[1]: Unmounted /var/cache/pacman/custom.
Apr 22 06:49:16 anystation systemd[1]: var-cache-pacman-pkg.mount: Deactivated successfully.
Apr 22 06:49:16 anystation systemd[1]: Unmounted /var/cache/pacman/pkg.
Apr 22 06:49:16 anystation systemd[1]: var-lib-libvirt.mount: Mount process exited, code=exited, status=32/n/a
Apr 22 06:49:16 anystation systemd[1]: Failed unmounting /var/lib/libvirt.

Bogus device unit states after replace operationL

$ systemctl status dev-mapper-new.device
* dev-mapper-new.device - /dev/mapper/new
     Loaded: loaded
     Active: activating (tentative) since Thu 2021-04-22 06:49:16 MSK; 9min ago
     Device: /sys/devices/virtual/block/dm-4

Apr 22 06:51:44 anystation systemd[1]: Unnecessary job for /dev/mapper/new was removed.

$ systemctl status dev-disk-by\\x2dlabel-anystation.device
* dev-disk-by\x2dlabel-anystation.device - /dev/disk/by-label/anystation
     Loaded: loaded
     Active: inactive (dead) since Thu 2021-04-22 06:49:16 MSK; 14min ago

Apr 22 03:52:28 archlinux systemd[1]: Found device /dev/disk/by-label/anystation.
Apr 22 06:51:44 anystation systemd[1]: dev-disk-by\x2dlabel-anystation.device: Job dev-disk-by\x2dlabel-anystation.device/start timed out.
Apr 22 06:51:44 anystation systemd[1]: Timed out waiting for device /dev/disk/by-label/anystation.
Apr 22 06:51:44 anystation systemd[1]: dev-disk-by\x2dlabel-anystation.device: Job dev-disk-by\x2dlabel-anystation.device/start failed with result 'timeout'.

Udev state dump of the replace target:

$ sudo udevadm info /dev/mapper/new
P: /devices/virtual/block/dm-4
N: dm-4
L: 0
S: disk/by-id/dm-uuid-CRYPT-LUKS2-18350e9905aa463fa357546d611ec19b-new
S: disk/by-id/dm-name-new
S: mapper/new
E: DEVPATH=/devices/virtual/block/dm-4
E: DEVNAME=/dev/dm-4
E: DEVTYPE=disk
E: MAJOR=254
E: MINOR=4
E: SUBSYSTEM=block
E: USEC_INITIALIZED=4773594254
E: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1
E: DM_UDEV_PRIMARY_SOURCE_FLAG=1
E: DM_UDEV_RULES_VSN=2
E: DM_NAME=new
E: DM_UUID=CRYPT-LUKS2-18350e9905aa463fa357546d611ec19b-new
E: DM_SUSPENDED=0
E: SYSTEMD_READY=0
E: DEVLINKS=/dev/disk/by-id/dm-uuid-CRYPT-LUKS2-18350e9905aa463fa357546d611ec19b-new /dev/disk/by-id/dm-name-new /dev/mapper/new
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

[cmurf]

Seems similar to #14674