work around failure of udev to observe btrfs device add #1606

Closed
phillxnet opened this Issue Jan 9, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@phillxnet
Member

phillxnet commented Jan 9, 2017

In some rare instances it is possible for udev and lsblk to fail to reflect the current state of a pools disk members. This was initially observed while testing btrfs in partition device members in issue #1494, however I have now duplicated the same failure using whole disk members.

So far this has been observed (sporadically) only directly after adding a device to a pool, but only after a few such operations have taken place just prior to the issue related device add. The most reliable indicator was still very intermittent but essentially involved adding and then removing and re-adding the same device to a pool, often multiple times.

The fail state is identified by lsblk and udev info not having been updated directly after the following command:

btrfs device add devname mountpoint

The failed state is then persistent. It was found the ‘btrfs device scan’ had no effect but:

echo change > /sys/block/vdd/vdd2/uevent

where vdd2 (a partitioned pool device member) was added but not accounted for in lsblk or udev info.
However on subsequent issue reproduction I was also able to test a system wide udev trigger as workaround:

udevadm trigger

Which also succeeded in returning udev and lsblk output to what would normally be expected, ie in line with the output of btrfs fi show.

Given Rockstor reliance on udev to maintain a current view on devices the resulting failed state causes the most recently added device to not be listed or recognised as a pool member.

Examples of failed / inconsistent state between btrfs fi show and udev / lsblk.
Here we see a 2 device pool where udev only knows of one of it's member.
The first pool example was created via:
mkfs.btrfs -f -d single -m single -L testpool /dev/vdd2 /dev/vdc2
and after multiple add and delete commands for one and then the other device we have:

Label: 'testpool'  uuid: d8357c81-1352-43f9-8ab3-883a9cd1cc03
	Total devices 2 FS bytes used 256.00KiB
	devid    2 size 2.85GiB used 592.00MiB path /dev/vdc2
	devid    3 size 2.85GiB used 0.00B path /dev/vdd2

But an lsblk output that fails to show /dev/vdd2 (devid 3 in above):

NAME="vdc2" MODEL="" SERIAL="" SIZE="2.9G" TRAN="" VENDOR="" HCTL="" TYPE="part" FSTYPE="btrfs" LABEL="testpool" UUID="d8357c81-1352-43f9-8ab3-883a9cd1cc03"

Also note that udevadm info --name /dev/vdd2 | grep FS_ had no output.
where as it's counterpart /dev/vdc2 show the expected btrfs FS_ entries.

In the above case the "echo change > /sys/block/vdd/vdd2/uevent" was found to resolve the inconsistency.

Another example involved a 2 device pool having no btrfs info reflected by udev and lsblk for either of it's members:

Label: 'testpool'  uuid: d31d8295-3fa6-4555-8ef2-ab72a7a5f828
	Total devices 2 FS bytes used 256.00KiB
	devid    2 size 2.85GiB used 592.00MiB path /dev/vdd2
	devid    3 size 2.85GiB used 0.00B path /dev/vdc2

where "lsblk -a -P -o NAME,TYPE,FSTYPE,LABEL,UUID | grep testpool"
had no output ie without the grep we see the member info as empy:

NAME="vdd2" TYPE="part" FSTYPE="" LABEL="" UUID=""
NAME="vdc2" TYPE="part" FSTYPE="" LABEL="" UUID=""

Another example involved a single device pool where that device pool member had no btrfs related info in udev or lsblk:

btrfs fi show testpool
Label: 'testpool'  uuid: d31d8295-3fa6-4555-8ef2-ab72a7a5f828
	Total devices 1 FS bytes used 256.00KiB
	devid    3 size 2.85GiB used 592.00MiB path /dev/vdc2

and again no appropriate btrfs or label info found in lsblk's output yet we have an active mount:

mount | grep testpool
/dev/vdc2 on /mnt2/testpool type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)

In this case I was able to test the "udevadm trigger" command and then we see:

lsblk -a -P -o NAME,TYPE,FSTYPE,LABEL,UUID | grep testpool
NAME="vdc2" TYPE="part" FSTYPE="btrfs" LABEL="testpool" UUID="d31d8295-3fa6-4555-8ef2-ab72a7a5f828"

So we have our return lsblk info on btrfs.
and our FS_ entries in udev also re-appeared:

udevadm info --name /dev/vdc2 | grep FS_
E: ID_FS_LABEL=testpool
E: ID_FS_LABEL_ENC=testpool
E: ID_FS_TYPE=btrfs
E: ID_FS_USAGE=filesystem
E: ID_FS_UUID=d31d8295-3fa6-4555-8ef2-ab72a7a5f828
E: ID_FS_UUID_ENC=d31d8295-3fa6-4555-8ef2-ab72a7a5f828
E: ID_FS_UUID_SUB=a716ba75-b8b9-4403-8be0-f85ede430af6
E: ID_FS_UUID_SUB_ENC=a716ba75-b8b9-4403-8be0-f85ede430af6

where they had previously been absent.

Finally I was able to catch a full disk example:

btrfs fi show testpool
Label: 'testpool'  uuid: 03cf9a94-4d88-4179-b3a9-0f53e22682c1
	Total devices 2 FS bytes used 256.00KiB
	devid    5 size 2.00GiB used 448.00MiB path /dev/vde
	devid    6 size 2.20GiB used 0.00B path /dev/vdf

And the lsblk output for the command:

lsblk -a -P -o NAME,TYPE,FSTYPE,LABEL,UUID | grep testpool

was:

NAME="vde" TYPE="disk" FSTYPE="btrfs" LABEL="testpool" UUID="03cf9a94-4d88-4179-b3a9-0f53e22682c1"

ie no entry for /dev/vdf

When at the same time we see no output from "udevadm info --name /dev/vdf | grep FS_"
only expected info is for the other /dev/vde member.

In this example the "udevadm trigger" command also succeeded in resolving these discrepancies.
@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jan 9, 2017

Member

I am now preparing a pr to address this issue.

Member

phillxnet commented Jan 9, 2017

I am now preparing a pr to address this issue.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jan 9, 2017

add 'udevadm trigger' wrapper and call after device add #1606
This was found to be required to ensure lsblk and udev info
was successfully updated after a btrfs device add command.
The failed state of stale info was only rarely observed but
was reproduced a number of times using both whole and
partition pool members.
@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jan 10, 2017

Member

Thank to maxhq in the following forum thread for also confirming the same behaviour can be seen after the:

btrfs replace start olddisk newdisk

command which also adds a disk but also removes another. The same 'udevadm trigger' command succeeded in restoring the expected udev/lsblk output in that case:
https://forum.rockstor.com/t/problems-with-disk-replacement/2660
In that case the added disk was once again the one missing in at least the lsblk output.

Member

phillxnet commented Jan 10, 2017

Thank to maxhq in the following forum thread for also confirming the same behaviour can be seen after the:

btrfs replace start olddisk newdisk

command which also adds a disk but also removes another. The same 'udevadm trigger' command succeeded in restoring the expected udev/lsblk output in that case:
https://forum.rockstor.com/t/problems-with-disk-replacement/2660
In that case the added disk was once again the one missing in at least the lsblk output.

@schakrava schakrava closed this in #1607 Jan 15, 2017

schakrava added a commit that referenced this issue Jan 15, 2017

Merge pull request #1607 from phillxnet/1606_work_around_failure_of_u…
…dev_to_observe_btrfs_device_add

work around failure of udev to observe btrfs device add. Fixes #1606

@schakrava schakrava added the bug label Jan 15, 2017

@schakrava schakrava added this to the Pinnacles milestone Jan 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment