udev hook (udevadm/systemd-udevd) Unable to active thin device while pool is suspended

[jamespharvey20]

Submission type

• Bug report
• Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

v231

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

No errors during ":: Triggering uevents"

In case of bug report: Unexpected behaviour you saw

:: Triggering uevents...
[ 3.106579] device-mapper: table: 253:7: thin: Unable to
activate thin device while pool is suspended
[ 3.290255] device-mapper: table: 253:17: thin: Unable to
activate thin device while pool is suspended
. . .

In case of bug report: Steps to reproduce the problem

At the bottom is a list of the minimal steps I've found that reproduces the problem. If I remove some of the lvcreates that aren't being used in these minimal steps, the problem seems to happen less often. So I'm not sure if that has an affect on the race condition, or if I'm just seeing variance, so I've left those in there even though they don't appear to do anything.

No idea if this is a udevadm bug, or a device mapper/kernel bug.

I think udevadm (or something it starts, dmesg shows systemd-udevd) is trying to activate a device as a thin device, when it isn't one. Or, this gets retried later because the system appears to function. But I'm certainly worried if there's a ticking timebomb, so I'm not wanting to just ignore it.

It happens during the udev hook while running "udevadm trigger --action=add --type=devices". (Editing the systemd udev hook, inserting sleep's between udevadm calls shows this - without the sleeps it shows during "udevadm settle" is running, since the trigger call is running in the background.) It happens very often, but not always.

Up to date Arch, linux 4.6.4, lvm2 2.02.162, systemd 231.

Appears to be a race condition. The error is given with different numbers. The number of errors (sometimes zero) varies too.

I just noticed "dmsetup ls" and "dmsetup table" are showing different 253:X numbers. I don't know which version of the numbers are showing during these dmesg errors in the initramfs.

If the error corresponds to the "dmsetup ls" numbers, than the error in this current boot refers to "disk2-disk2thin_tdata" and "disk3-persistent3", which is ext4. Over an mdadm array, the error occurs, and on a single lv it doesn't occur.

If the error corresponds to the "dmsetup table" numbers, then the error refers to a device not in the table as well as possibly "disk2-disk2thin-tpool".

If during the install I skip making the mdadm boot array and just use a single partition, this doesn't happen.

Attached is a "lvmdump -m" file. And a full dmesg through shutdown.

lvmdump-terra-2016080534140.zip
udevError.shutdown-log.txt

$ sudo dmsetup ls
disk3-disk3thin (253:15)
disk2-main2 (253:10)
disk3-disk3thin-tpool (253:14)
disk3-disk3thin_tdata (253:13)
disk1-disk1thin-tpool (253:2)
disk1-disk1thin_tdata (253:1)
disk3-disk3thin_tmeta (253:12)
disk3-main3 (253:16)
disk2-disk2thin (253:9)
disk1-disk1thin_tmeta (253:0)
disk2-persistent2 (253:11)
disk1-disk1thin (253:3)
disk2-disk2thin-tpool (253:8)
disk2-disk2thin_tdata (253:7)
disk1-persistent1 (253:5)
disk2-disk2thin_tmeta (253:6)
disk1-main1 (253:4)
disk3-persistent3 (253:17)

$ sudo dmsetup table

disk3-disk3thin: 0 1048576000 linear 253:14 0
disk2-main2: 0 209715200 thin 253:8 1
disk3-disk3thin-tpool: 0 1048576000 thin-pool 253:12 253:13 512 0 0
disk3-disk3thin_tdata: 0 1048576000 linear 8:35 264192
disk1-disk1thin-tpool: 0 1048576000 thin-pool 253:0 253:1 512 0 0
disk1-disk1thin_tdata: 0 1048576000 linear 8:3 264192
disk3-disk3thin_tmeta: 0 262144 linear 8:35 1048840192
disk3-main3: 0 209715200 thin 253:14 1
disk2-disk2thin: 0 1048576000 linear 253:8 0
disk1-disk1thin_tmeta: 0 262144 linear 8:3 1048840192
disk2-persistent2: 0 209715200 thin 253:8 2
disk1-disk1thin: 0 1048576000 linear 253:2 0
disk2-disk2thin-tpool: 0 1048576000 thin-pool 253:6 253:7 512 0 0
disk2-disk2thin_tdata: 0 1048576000 linear 8:19 264192
disk1-persistent1: 0 209715200 thin 253:2 2
disk2-disk2thin_tmeta: 0 262144 linear 8:19 1048840192
disk1-main1: 0 209715200 thin 253:2 1
disk3-persistent3: 0 209715200 thin 253:14 2

$ sudo lvmdump -m
Creating dump directory: /root/lvmdump-terra-2016080534140

Gathering LVM & device-mapper version info...
Gathering dmsetup info...
Gathering process info...
Gathering console messages...
Gathering /etc/lvm info...
Gathering /dev listing...
Gathering /sys/block listing...
Gathering LVM metadata from Physical Volumes...
/dev/sda3
/dev/sdb3
/dev/sdc3
Creating report tarball in /root/lvmdump-terra-2016080534140.tgz...

/dev/sd{a,b,c}1 3.5G Linux RAID
/dev/sd{a,b,c}2 3.5G Linux RAID
/dev/sd{a,b,c}3 4.5T Linux LVM

{ Setup LVM and filesystems }

({ This causes the issue })
mdadm --create --verbose --name=main_boot --homehost="none" --level 1 --metadata 1.0 --raid-devices=3 /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1
mkfs.ext4 -F -L main_boot /dev/disk/by-id/md-name-main_boot

({ This appears to prevent the issue })
mkfs.ext4 -F -L main_boot /dev/sda

({ Then, either way, continuing })
pvcreate --yes --pvmetadatacopies 2 /dev/sda3
vgcreate disk1 /dev/sda3
pvcreate --yes --pvmetadatacopies 2 /dev/sdb3
vgcreate disk2 /dev/sdb3
pvcreate --yes --pvmetadatacopies 2 /dev/sdc3
vgcreate disk3 /dev/sdc3
lvcreate --size 500G --thinpool disk1thin disk1
lvcreate --size 500G --thinpool disk2thin disk2
lvcreate --size 500G --thinpool disk3thin disk3
lvcreate --virtualsize 100G --name main1 disk1/disk1thin
lvcreate --virtualsize 100G --name main2 disk2/disk2thin
lvcreate --virtualsize 100G --name main3 disk3/disk3thin
mkfs.ext4 -L main /dev/disk1/main1
mount /dev/disk1/main1 /mnt
mkdir /mnt/boot
mount /dev/dev/sda1 /mnt/boot
lvcreate --virtualsize 100G --name persistent1 disk1/disk1thin
lvcreate --virtualsize 100G --name persistent2 disk2/disk2thin
lvcreate --virtualsize 100G --name persistent3 disk3/disk3thin

{ Install Arch Linux}
vi /etc/pacman.d/mirrorlist
pacstrap -i base syslinux gptfdisk lvm2
arch-chroot /mnt
vi /etc/locale.gen
locale-gen
locale > /etc/locale.conf

({ If using the mdadm to cause the error })
mdadm --detail --scan > /etc/mdadm.conf

({ Then, either way, continuing })
vi /etc/nsswitch.conf
systemd enable systemd-resolved systemd-networkd
ln -s /usr/share/zoneinfo/America/Detroit /etc/localtime
hwclock --utc --systohc
passwd
{ Add lvm2 between block and filesystems}
vi /etc/mkinitcpio.conf
mkinitcpio -p linux
echo hostname > /etc/hostname
vi /etc/systemd/network/enp31s0.network
syslinux-install_update -i -a -m
vi /boot/syslinux/syslinux.cfg
{{ After exiting the arch-chroot }}
ln -f -s /run/systemd/resolve/resolv.conf /mnt/etc/resolv.conf

{ Reboot }

[poettering]

Sorry, but we know nothing about LVM/DM. They install their own udev rules that divert a good chunk of the usual control flow. if you have issues with LVM/DM please contact the LVM/DM people, we can't do much about that. Sorry!