New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

insufficient use of btrfs device scan #1547

Closed
phillxnet opened this Issue Nov 27, 2016 · 3 comments

Comments

Projects
None yet
2 participants
@phillxnet
Member

phillxnet commented Nov 27, 2016

Currently we only execute this command on boot via bootstrap.py btrfs/device_scan(). In certain, currently rare, circumstance this can lead to failed mounts with multi-volume filesystems when mounting by label as we do: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#Filesystem_can.27t_be_mounted_by_label

"... if one volume of a multi-volume filesystem fails when mounting, but the other succeeds:
# mount /dev/sda1 /mnt/fs
mount: wrong fs type, bad option, bad superblock on /dev/sdd2,
      missing codepage or helper program, or other error
      In some cases useful info is found in syslog - try
      dmesg | tail  or so
# mount /dev/sdb1 /mnt/fs
#

Then you need to ensure that you run a btrfs device scan first:
# btrfs device scan

Rockstor log excerpt of the same:

[27/Nov/2016 16:45:29] ERROR [storageadmin.views.command:87] Exception while refreshing state for Pool(luks-pool-on-bcache). Moving on: Error running a command. cmd = ['/bin/mount', '/dev/disk/by-label/luks-pool-on-bcache', '/mnt2/luks-pool-on-bcache', '-o', ',compress=no']. rc = 32. stdout = ['']. stderr = ['mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,', '       missing codepage or helper program, or other error', '', '       In some cases useful info is found in syslog - try', '       dmesg | tail or so.', '']

In the above instance a 2 disk LUKS full disk encrypted pool was unable to mount as it's drives had only been unlocked after boot.

Label: 'luks-pool-on-bcache'  uuid: bb027f88-a3d9-445c-b1d1-ffb86a103c98
	Total devices 2 FS bytes used 272.00KiB
	devid    1 size 2.00GiB used 417.12MiB path /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e
	devid    2 size 2.00GiB used 417.12MiB path /dev/mapper/luks-a47f4950-3296-4504-b9a4-2dc75681a6ad

And our "by-label" points to one of them as expected:

ls -la /dev/disk/by-label/luks-pool-on-bcache 
lrwxrwxrwx 1 root root 10 Nov 27 16:41 /dev/disk/by-label/luks-pool-on-bcache -> ../../dm-1

and we find:

cat /sys/devices/virtual/block/dm-1/dev 
251:1

with lsblk having output of:

sdb                                             8:16  bcache-bdev-1        disk  bcache            
└─bcache1                                     252:1                        disk  crypto_LUKS       
  └─luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e 251:1                        crypt btrfs             luks-pool-on-bcache

So 251:1 is our luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e

But if we try and mount via this device:

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

Note that the resolved mapper path is correct.
But if we attempt to mount the ‘other device’ listed in btrfs fi show:

/bin/mount /dev/mapper/luks-a47f4950-3296-4504-b9a4-2dc75681a6ad /mnt2/luks-pool-on-bcache/

then we are good and get a successful mount.

So if we are to move to a more dynamic disk model, ie such as external backup media or live plugging of disks, or occasional after boot LUKS container opening then we must first run our
btrfs/device_scan() function to refresh disk data prior to attempting mounts.

I am unsure where best this should be implemented.

Maybe in scan_disks() but that may just be over the top or better still just before each mount ie in:

mount_root(pool):
"""
Mounts a given pool at the default mount root (usually /mnt2/) ...
"""

@schakrava I suspect this one is more in your realm: I can easily confirm fix with my current role issue #1494 where I am trialling live disk / pool additions.

@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Nov 28, 2016

Member

In follow up confirmation testing it was found that this same pool failed to mount for an extended period even with multiple refreshes (causing the same log output as before) of the pools page but once:

btrfs device scan
Scanning for Btrfs filesystems

was executed the pool was there after auto mounted successfully while the pool page was visible.

Member

phillxnet commented Nov 28, 2016

In follow up confirmation testing it was found that this same pool failed to mount for an extended period even with multiple refreshes (causing the same log output as before) of the pools page but once:

btrfs device scan
Scanning for Btrfs filesystems

was executed the pool was there after auto mounted successfully while the pool page was visible.

@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jun 11, 2017

Member

I am currently having another look at this issue.

Member

phillxnet commented Jun 11, 2017

I am currently having another look at this issue.

@phillxnet

This comment has been minimized.

Show comment
Hide comment
@phillxnet

phillxnet Jun 17, 2017

Member

As a system wide btrfs device scan (all block devices) seems over kill prior to every pool mount; the above failure to mount by label and it's consequent disk was reproduced (note that this in an intermittent issue): and then only pool members were scanned prior to a pool mount attempt ie:

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

where:

ls -la /dev/disk/by-label/luks-pool-on-bcache 
lrwxrwxrwx 1 root root 10 Jun 17 10:52 /dev/disk/by-label/luks-pool-on-bcache -> ../../dm-1

and dm-1 in turn links to luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e

cat /sys/devices/virtual/block/dm-1/dev
251:1

and

sdd                                             8:48   0    2G  0 disk  
├─bcache0                                     252:0    0    2G  0 disk  
│ └─luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e 251:1 

Then we do a device specific btrfs scan of our first pool device:

btrfs device scan /dev/disk/by-id/dm-name-luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e 
Scanning for Btrfs filesystems in '/dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e'

and re-test our mount (which again fails):

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

then we do a device specific device scan of our second pool device:

btrfs device scan /dev/disk/by-id/dm-name-luks-a47f4950-3296-4504-b9a4-2dc75681a6ad 
Scanning for Btrfs filesystems in '/dev/mapper/luks-a47f4950-3296-4504-b9a4-2dc75681a6ad'

and then retest our mount by label again (which this time succeeds):

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/

So we now have a successful mount by label after first device scanning each member:

mount | grep luks-pool
/dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e on /mnt2/luks-pool-on-bcache type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)
Member

phillxnet commented Jun 17, 2017

As a system wide btrfs device scan (all block devices) seems over kill prior to every pool mount; the above failure to mount by label and it's consequent disk was reproduced (note that this in an intermittent issue): and then only pool members were scanned prior to a pool mount attempt ie:

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

where:

ls -la /dev/disk/by-label/luks-pool-on-bcache 
lrwxrwxrwx 1 root root 10 Jun 17 10:52 /dev/disk/by-label/luks-pool-on-bcache -> ../../dm-1

and dm-1 in turn links to luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e

cat /sys/devices/virtual/block/dm-1/dev
251:1

and

sdd                                             8:48   0    2G  0 disk  
├─bcache0                                     252:0    0    2G  0 disk  
│ └─luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e 251:1 

Then we do a device specific btrfs scan of our first pool device:

btrfs device scan /dev/disk/by-id/dm-name-luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e 
Scanning for Btrfs filesystems in '/dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e'

and re-test our mount (which again fails):

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

then we do a device specific device scan of our second pool device:

btrfs device scan /dev/disk/by-id/dm-name-luks-a47f4950-3296-4504-b9a4-2dc75681a6ad 
Scanning for Btrfs filesystems in '/dev/mapper/luks-a47f4950-3296-4504-b9a4-2dc75681a6ad'

and then retest our mount by label again (which this time succeeds):

/bin/mount /dev/disk//by-label/luks-pool-on-bcache /mnt2/luks-pool-on-bcache/

So we now have a successful mount by label after first device scanning each member:

mount | grep luks-pool
/dev/mapper/luks-3efb3830-fee1-4a9e-a5c6-ea456bfc269e on /mnt2/luks-pool-on-bcache type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/)

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 19, 2017

add Disk.target_name model method / property #1547
This model extension eases access to a redirect role
name if one exists. Fail over is to the whole disk
name.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 19, 2017

revise _refresh_pool_state() to use Disk.target_name #1547
Reduce code duplication by using recent Disk model enhancement.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 19, 2017

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 19, 2017

pre btrfs scan all pool members prior to mount #1547
To avoid a system wide 'btrfs device scan' prior to every
pool mount we individually scan each pool member. On
initial boot this is redundant but allows for a more dynamic
and robust pool mount mechanism going forward.
Includes unit tests to cover the device_scan wrapper
enhancements.

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Jun 19, 2017

fix bug in parted dev share mount via Disk.target_name #1547
As mount_share / mount_snap directly access disk.name they
will fail if the first device in the pool has a redirect
role. Resolve by using recent Disk.target_name model
enhancement.

schakrava added a commit that referenced this issue Jun 24, 2017

add Disk.target_name model method / property #1547
This model extension eases access to a redirect role
name if one exists. Fail over is to the whole disk
name.

schakrava added a commit that referenced this issue Jun 24, 2017

revise _refresh_pool_state() to use Disk.target_name #1547
Reduce code duplication by using recent Disk model enhancement.

schakrava added a commit that referenced this issue Jun 24, 2017

pre btrfs scan all pool members prior to mount #1547
To avoid a system wide 'btrfs device scan' prior to every
pool mount we individually scan each pool member. On
initial boot this is redundant but allows for a more dynamic
and robust pool mount mechanism going forward.
Includes unit tests to cover the device_scan wrapper
enhancements.

schakrava added a commit that referenced this issue Jun 24, 2017

fix bug in parted dev share mount via Disk.target_name #1547
As mount_share / mount_snap directly access disk.name they
will fail if the first device in the pool has a redirect
role. Resolve by using recent Disk.target_name model
enhancement.

@schakrava schakrava closed this in #1737 Jun 24, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment