SLES12 with no snapshots failed to recreate/mount btrfs FS during recovery. #1435

schabrolles · 2017-07-28T18:55:35Z

If you install SLES12 with btrfs support but without snapshots you should face some issue during recovery:

During layout creation, ReaR will fail to mount btrfs FS (because subvolumeID is empty)

+++ subvolumeID=
+++ btrfs subvolume set-default /mnt/local/
btrfs subvolume set-default: too few arguments
usage: btrfs subvolume set-default <subvolid> <path>

    Set the default subvolume of a filesystem

2017-07-25 17:33:28.564682686 An error occurred during layout recreation.

Looking at disklayout.conf shows that NO btrfsnormalsubvol are present (=> no subvol creation)

# Disk /dev/sda
# Format: disk <devname> <size(bytes)> <partition label type>
disk /dev/sda 53687091200 msdos
# Partitions on /dev/sda
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>
part /dev/sda 7340032 1048576 primary boot,prep /dev/sda1
part /dev/sda 53678702592 8388608 primary lvm /dev/sda2
lvmdev /dev/system /dev/sda2 m5HZEO-DWBB-ShHr-YZgm-sloR-8V9Q-aRnFm1 104841216
lvmgrp /dev/system 4096 12797 52416512
lvmvol /dev/system root 10240 83886080 
lvmvol /dev/system swap 512 4194304 
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]
fs /dev/mapper/system-root / btrfs uuid=681502da-fcba-4da8-8925-b538f4c09049 label= options=rw,relatime,space_cache,subvolid=257,subvol=/@
# Btrfs default subvolume for /dev/mapper/system-root at /
# Format: btrfsdefaultsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>
btrfsdefaultsubvol /dev/mapper/system-root / 257 @
# Btrfs normal subvolumes for /dev/mapper/system-root at /
# Format: btrfsnormalsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>
# All mounted btrfs subvolumes (including mounted btrfs default subvolumes and mounted btrfs snapshot subvolumes).
# Determined by the findmnt command that shows the mounted btrfs_subvolume_path.
# Format: btrfsmountedsubvol <device> <subvolume_mountpoint> <mount_options> <btrfs_subvolume_path>
btrfsmountedsubvol /dev/mapper/system-root / rw,relatime,space_cache,subvolid=257,subvol=/@ @
btrfsmountedsubvol /dev/mapper/system-root /boot/grub2/powerpc-ieee1275 rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/powerpc-ieee1275 @/boot/grub2/powerpc-ieee1275
btrfsmountedsubvol /dev/mapper/system-root /home rw,relatime,space_cache,subvolid=259,subvol=/@/home @/home
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mailman rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mailman @/var/lib/mailman
btrfsmountedsubvol /dev/mapper/system-root /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash @/var/crash
btrfsmountedsubvol /dev/mapper/system-root /opt rw,relatime,space_cache,subvolid=260,subvol=/@/opt @/opt
btrfsmountedsubvol /dev/mapper/system-root /var/cache rw,relatime,space_cache,subvolid=264,subvol=/@/var/cache @/var/cache
btrfsmountedsubvol /dev/mapper/system-root /var/spool rw,relatime,space_cache,subvolid=275,subvol=/@/var/spool @/var/spool
btrfsmountedsubvol /dev/mapper/system-root /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images @/var/lib/libvirt/images
btrfsmountedsubvol /dev/mapper/system-root /srv rw,relatime,space_cache,subvolid=261,subvol=/@/srv @/srv
btrfsmountedsubvol /dev/mapper/system-root /var/log rw,relatime,space_cache,subvolid=273,subvol=/@/var/log @/var/log
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mysql rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/mysql @/var/lib/mysql
btrfsmountedsubvol /dev/mapper/system-root /tmp rw,relatime,space_cache,subvolid=262,subvol=/@/tmp @/tmp
btrfsmountedsubvol /dev/mapper/system-root /var/lib/named rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/named @/var/lib/named
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mariadb rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mariadb @/var/lib/mariadb
btrfsmountedsubvol /dev/mapper/system-root /var/lib/machines rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/machines @/var/lib/machines
btrfsmountedsubvol /dev/mapper/system-root /usr/local rw,relatime,space_cache,subvolid=263,subvol=/@/usr/local @/usr/local
btrfsmountedsubvol /dev/mapper/system-root /var/lib/pgsql rw,relatime,space_cache,subvolid=272,subvol=/@/var/lib/pgsql @/var/lib/pgsql
btrfsmountedsubvol /dev/mapper/system-root /var/opt rw,relatime,space_cache,subvolid=274,subvol=/@/var/opt @/var/opt
btrfsmountedsubvol /dev/mapper/system-root /var/tmp rw,relatime,space_cache,subvolid=276,subvol=/@/var/tmp @/var/tmp
# Mounted btrfs subvolumes that have the 'no copy on write' attribute set.
# Format: btrfsnocopyonwrite <btrfs_subvolume_path>
btrfsnocopyonwrite @/var/lib/libvirt/images
btrfsnocopyonwrite @/var/lib/mysql
btrfsnocopyonwrite @/var/lib/mariadb
btrfsnocopyonwrite @/var/lib/pgsql
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/mapper/system-swap uuid=890ff8c3-e4ca-4189-8117-f128e72e1a80 label=

This seems to come from usr/share/rear/layout/save/GNU/Linux/230_filesystem_layout.sh
SLES12 without snapshot controlled by snapper will generate an empty $snapshot_subvolumes_pattern variable.

snapshot_subvolumes_pattern=$( btrfs subvolume list -as $btrfs_mountpoint | tr -s '[:blank:]' ' ' | cut -d ' ' -f 2 | sed -e 's/^/^/' -e 's/$/ |/' | tr -d '\n' | sed -e 's/|$//' )

because of that, $subvolumes_exclude_pattern variable generated looks like this:
subvolumes_exclude_pattern="|@.snapshots"
=> This will filter ALL the subvolumes listed by btrfs subvolume list -a /

+++ btrfs subvolume list -as /
+++ tr -s '[:blank:]' ' '
+++ cut -d ' ' -f 2
+++ sed -e 's/^/^/' -e 's/$/ |/'
+++ tr -d '\n'
+++ sed -e 's/|$//'
++ snapshot_subvolumes_pattern=
++ snapper_base_subvolume=@/.snapshots
++ subvolumes_exclude_pattern='|@/.snapshots'
++ echo '# Btrfs normal subvolumes for /dev/mapper/system-root at /'
++ echo '# Format: btrfsnormalsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>'
++ btrfs subvolume list -a /
++ grep -q @/.snapshots
++ test -z '|@/.snapshots'
++ echo '257 @
258 @/boot/grub2/powerpc-ieee1275
259 @/home
260 @/opt
261 @/srv
262 @/tmp
263 @/usr/local
264 @/var/cache
265 @/var/crash
266 @/var/lib/libvirt/images
267 @/var/lib/machines
268 @/var/lib/mailman
269 @/var/lib/mariadb
270 @/var/lib/mysql
271 @/var/lib/named
272 @/var/lib/pgsql
273 @/var/log
274 @/var/opt
275 @/var/spool
276 @/var/tmp'
++ egrep -v '|@/.snapshots'
++ sed -e 's/^/btrfsnormalsubvol \/dev\/mapper\/system-root \/ /'
++ read_mounted_btrfs_subvolumes_command='mount -t btrfs | cut -d '\'' '\'' -f 1,3,6'
++ test -x /usr/bin/findmnt
++ /usr/bin/findmnt -mnrv -o FSROOT -t btrfs

We just need to test if the $snapshot_subvolumes_pattern is empty before setting the $subvolumes_exclude_pattern

Results after patch: (disklayout.conf) => (btrfsnormalsubvol are present before btrfsmountedsubvol)

# Disk /dev/sda                                                                                                                            
# Format: disk <devname> <size(bytes)> <partition label type>                                                                              
disk /dev/sda 53687091200 msdos                                                                                                            
# Partitions on /dev/sda                                                                                                                   
# Format: part <device> <partition size(bytes)> <partition start(bytes)> <partition type|name> <flags> /dev/<partition>                    
part /dev/sda 7340032 1048576 primary boot,prep /dev/sda1                                                                                  
part /dev/sda 53678702592 8388608 primary lvm /dev/sda2                                                                                     
lvmdev /dev/system /dev/sda2 m5HZEO-DWBB-ShHr-YZgm-sloR-8V9Q-aRnFm1 104841216                                                                
lvmgrp /dev/system 4096 12797 52416512                                                                                                        
lvmvol /dev/system root 10240 83886080                                                                                                        
lvmvol /dev/system swap 512 4194304                                                                                                            
# Filesystems (only ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs are supported).                                                                      
# Format: fs <device> <mountpoint> <fstype> [uuid=<uuid>] [label=<label>] [<attributes>]                                                                                                                                                      
fs /dev/mapper/system-root / btrfs uuid=681502da-fcba-4da8-8925-b538f4c09049 label= options=rw,relatime,space_cache,subvolid=257,subvol=/@                                                                                                    
# Btrfs default subvolume for /dev/mapper/system-root at /                                                                                                                                                                                    
# Format: btrfsdefaultsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>                                                                                                                                                
btrfsdefaultsubvol /dev/mapper/system-root / 257 @                                                                                                                                                                                            
# Btrfs normal subvolumes for /dev/mapper/system-root at /                                                                                                                                                                                    
# Format: btrfsnormalsubvol <device> <mountpoint> <btrfs_subvolume_ID> <btrfs_subvolume_path>                                                                                                                                                 
btrfsnormalsubvol /dev/mapper/system-root / 257 @                                                                                                                                                                                             
btrfsnormalsubvol /dev/mapper/system-root / 258 @/boot/grub2/powerpc-ieee1275                                                                                                                                                                 
btrfsnormalsubvol /dev/mapper/system-root / 259 @/home
btrfsnormalsubvol /dev/mapper/system-root / 260 @/opt
btrfsnormalsubvol /dev/mapper/system-root / 261 @/srv
btrfsnormalsubvol /dev/mapper/system-root / 262 @/tmp
btrfsnormalsubvol /dev/mapper/system-root / 263 @/usr/local
btrfsnormalsubvol /dev/mapper/system-root / 264 @/var/cache
btrfsnormalsubvol /dev/mapper/system-root / 265 @/var/crash
btrfsnormalsubvol /dev/mapper/system-root / 266 @/var/lib/libvirt/images
btrfsnormalsubvol /dev/mapper/system-root / 267 @/var/lib/machines
btrfsnormalsubvol /dev/mapper/system-root / 268 @/var/lib/mailman
btrfsnormalsubvol /dev/mapper/system-root / 269 @/var/lib/mariadb
btrfsnormalsubvol /dev/mapper/system-root / 270 @/var/lib/mysql
btrfsnormalsubvol /dev/mapper/system-root / 271 @/var/lib/named
btrfsnormalsubvol /dev/mapper/system-root / 272 @/var/lib/pgsql
btrfsnormalsubvol /dev/mapper/system-root / 273 @/var/log
btrfsnormalsubvol /dev/mapper/system-root / 274 @/var/opt
btrfsnormalsubvol /dev/mapper/system-root / 275 @/var/spool
btrfsnormalsubvol /dev/mapper/system-root / 276 @/var/tmp
# All mounted btrfs subvolumes (including mounted btrfs default subvolumes and mounted btrfs snapshot subvolumes).
# Determined by the findmnt command that shows the mounted btrfs_subvolume_path.
# Format: btrfsmountedsubvol <device> <subvolume_mountpoint> <mount_options> <btrfs_subvolume_path>
btrfsmountedsubvol /dev/mapper/system-root / rw,relatime,space_cache,subvolid=257,subvol=/@ @
btrfsmountedsubvol /dev/mapper/system-root /boot/grub2/powerpc-ieee1275 rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/powerpc-ieee1275 @/boot/grub2/powerpc-ieee1275
btrfsmountedsubvol /dev/mapper/system-root /home rw,relatime,space_cache,subvolid=259,subvol=/@/home @/home
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mailman rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mailman @/var/lib/mailman
btrfsmountedsubvol /dev/mapper/system-root /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash @/var/crash
btrfsmountedsubvol /dev/mapper/system-root /opt rw,relatime,space_cache,subvolid=260,subvol=/@/opt @/opt
btrfsmountedsubvol /dev/mapper/system-root /var/cache rw,relatime,space_cache,subvolid=264,subvol=/@/var/cache @/var/cache
btrfsmountedsubvol /dev/mapper/system-root /var/spool rw,relatime,space_cache,subvolid=275,subvol=/@/var/spool @/var/spool
btrfsmountedsubvol /dev/mapper/system-root /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images @/var/lib/libvirt/images
btrfsmountedsubvol /dev/mapper/system-root /srv rw,relatime,space_cache,subvolid=261,subvol=/@/srv @/srv
btrfsmountedsubvol /dev/mapper/system-root /var/log rw,relatime,space_cache,subvolid=273,subvol=/@/var/log @/var/log
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mysql rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/mysql @/var/lib/mysql
btrfsmountedsubvol /dev/mapper/system-root /tmp rw,relatime,space_cache,subvolid=262,subvol=/@/tmp @/tmp
btrfsmountedsubvol /dev/mapper/system-root /var/lib/named rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/named @/var/lib/named
btrfsmountedsubvol /dev/mapper/system-root /var/lib/mariadb rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mariadb @/var/lib/mariadb
btrfsmountedsubvol /dev/mapper/system-root /var/lib/machines rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/machines @/var/lib/machines
btrfsmountedsubvol /dev/mapper/system-root /usr/local rw,relatime,space_cache,subvolid=263,subvol=/@/usr/local @/usr/local
btrfsmountedsubvol /dev/mapper/system-root /var/lib/pgsql rw,relatime,space_cache,subvolid=272,subvol=/@/var/lib/pgsql @/var/lib/pgsql
btrfsmountedsubvol /dev/mapper/system-root /var/opt rw,relatime,space_cache,subvolid=274,subvol=/@/var/opt @/var/opt
btrfsmountedsubvol /dev/mapper/system-root /var/tmp rw,relatime,space_cache,subvolid=276,subvol=/@/var/tmp @/var/tmp
# Mounted btrfs subvolumes that have the 'no copy on write' attribute set.
# Format: btrfsnocopyonwrite <btrfs_subvolume_path>
btrfsnocopyonwrite @/var/lib/libvirt/images
btrfsnocopyonwrite @/var/lib/mysql
btrfsnocopyonwrite @/var/lib/mariadb
btrfsnocopyonwrite @/var/lib/pgsql
# Swap partitions or swap files
# Format: swap <filename> uuid=<uuid> label=<label>
swap /dev/mapper/system-swap uuid=890ff8c3-e4ca-4189-8117-f128e72e1a80 label=

…es_exclude_pattern variable. - When running SLES without snapper, $snapshot_subvolumes_pattern will be empty which cause a $subvolumes_exclude_pattern="|@.snapshot" which will exclude everything!

gozora · 2017-07-28T19:05:08Z

Sweet, if this works (and I really believe it does) it should solve #1036 as well...

V.

schabrolles · 2017-07-28T19:43:23Z

@gozora, Yes, it's really looks like the same bug....

I've only tested it on SLE12 SP2, it could be great if you can confirm it solve your issue #1036 with SLE12 SP1.

schabrolles · 2017-07-29T07:45:34Z

@jsmeix, your feedback is really welcomed as it is related to SLES with btrfs.

gozora · 2017-07-30T10:31:36Z

Hello @schabrolles,

I just successfully run test restore of SLES12 SP1 with disabled snapper and no snapshots with your patch.

V.

Fixed typo in issue reference in comment and provide it as directly usabel full URL ( 'rear#1345' -> 'rear#1435' )

Align code with the code below. But now I do no longer understand what went wrong because I already tested in the code below for <pre> if test -z "$snapshot_subvolumes_pattern" ; then </pre> I only commit this one to make it clear what I mean. Currently I do not understand what goes on.

jsmeix · 2017-07-31T08:53:25Z

@schabrolles @gozora
I commited two changes:

First:
ab5a3c8
is only cosmetical (fixed wrong issue number in comment).

Second:
b90a381
mainly for you to have a closer look because
now the code looks duplicated because I already do test
in the code below for

if test -z "$snapshot_subvolumes_pattern" ; then

so that the test above from @schabrolles should
not be needed but somehow things go wrong
without the test above from @schabrolles
and currently I do not understand what actually
goes wrong here.

@schabrolles @gozora
could you have a closer look and help me to find out
what actually goes wrong here - somehow my test

if test -z "$snapshot_subvolumes_pattern" ; then

seems to not work as intended?

gozora · 2017-07-31T09:08:11Z

Hi @jsmeix

Just from plain look on your patch

 if test -z "$snapshot_subvolumes_pattern " ; then

Is the space in the end of "$snapshot_subvolumes_pattern " used on purpose?

V.

jsmeix · 2017-07-31T09:09:35Z

Always - really always - when I have to work with
that oversophisticated SUSE btrfs default structure
my poor mathematician's mind gets "knotted" :-(

Meanwhile even I can see how it had failed before:
It is the subsequent test

# Output btrfs normal subvolumes:
if test -z "$subvolumes_exclude_pattern" ; then

that does not work as intended in case of

subvolumes_exclude_pattern='|@/.snapshots'

so that the test above from @schabrolles
is needed to make that subsequent test
work as intended.

Accordingly I think everything is right with this pull request
but I will test it on SLES12-SP2 with the btrfs default structure
to verify that it still works there (which is the main use-case).

jsmeix · 2017-07-31T09:12:02Z

Aaaaaaaaaaaarrrrrrrrrrrrrrggggggggggggghhhhhhhhhhhhh!
One space too much issues also hit me!
@gozora
many thanks for your careful looking
even at the emptiness in my code!

Removed meaningful emptiness that is wrong (a.k.a. "one space too much" at a certain place) which would have let the would stuff fail again.

schabrolles · 2017-07-31T09:23:27Z

@jsmeix HeHeHe .... Glad to see I'm not the only one ;)

Use double quotes for "$snapper_base_subvolume" to be safer against possible blanks therein.

jsmeix

It still works for me on SLES12-SP2
with the default btrfs structure
that uses snapper snapshots
so that there are no regressions.

jsmeix · 2017-07-31T09:40:04Z

@schabrolles
many thanks for your careful analysis of the root issue
of #1036
and for your fix here!

@schabrolles @gozora
from my point of view everything looks right now
(I even understand my own code a bit better now ;-)
and if you agree I would be happy to merge it soon.

Check if $napshot_subvolumes_patter is empty before setting $subvolum…

bf74c3b

…es_exclude_pattern variable. - When running SLES without snapper, $snapshot_subvolumes_pattern will be empty which cause a $subvolumes_exclude_pattern="|@.snapshot" which will exclude everything!

Add explanation and comment

38e81f0

Minor correction in comment

bc2ffe3

jsmeix self-assigned this Jul 31, 2017

jsmeix added the bug The code does not do what it is meant to do label Jul 31, 2017

jsmeix added this to the ReaR v2.3 milestone Jul 31, 2017

jsmeix self-requested a review July 31, 2017 08:22

jsmeix added 2 commits July 31, 2017 10:34

Update 230_filesystem_layout.sh

ab5a3c8

Fixed typo in issue reference in comment and provide it as directly usabel full URL ( 'rear#1345' -> 'rear#1435' )

Update 230_filesystem_layout.sh

0c76f32

Removed meaningful emptiness that is wrong (a.k.a. "one space too much" at a certain place) which would have let the would stuff fail again.

Update 230_filesystem_layout.sh

dab81c5

Use double quotes for "$snapper_base_subvolume" to be safer against possible blanks therein.

jsmeix approved these changes Jul 31, 2017

View reviewed changes

jsmeix merged commit 97e0780 into rear:master Jul 31, 2017

jsmeix added the fixed / solved / done label Jul 31, 2017

jsmeix mentioned this pull request Jul 31, 2017

SLES12 SP1 recovery fails with btrfs if snapper is not enabled #1036

Closed

schabrolles deleted the sles_with_no_snapper branch August 11, 2017 17:19

jsmeix mentioned this pull request Mar 21, 2018

OpenSuse Leap 42.3 - recover fails gdha/rear-automated-testing#55

Closed

jsmeix mentioned this pull request May 14, 2018

No code has been generated to recreate pv:/dev/mapper/mpathc_part2 (lvmdev) #1796

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SLES12 with no snapshots failed to recreate/mount btrfs FS during recovery. #1435

SLES12 with no snapshots failed to recreate/mount btrfs FS during recovery. #1435

schabrolles commented Jul 28, 2017 •

edited

gozora commented Jul 28, 2017

schabrolles commented Jul 28, 2017

schabrolles commented Jul 29, 2017

gozora commented Jul 30, 2017

jsmeix commented Jul 31, 2017

gozora commented Jul 31, 2017

jsmeix commented Jul 31, 2017 •

edited

jsmeix commented Jul 31, 2017 •

edited

schabrolles commented Jul 31, 2017

jsmeix left a comment

jsmeix commented Jul 31, 2017

SLES12 with no snapshots failed to recreate/mount btrfs FS during recovery. #1435

SLES12 with no snapshots failed to recreate/mount btrfs FS during recovery. #1435

Conversation

schabrolles commented Jul 28, 2017 • edited

gozora commented Jul 28, 2017

schabrolles commented Jul 28, 2017

schabrolles commented Jul 29, 2017

gozora commented Jul 30, 2017

jsmeix commented Jul 31, 2017

gozora commented Jul 31, 2017

jsmeix commented Jul 31, 2017 • edited

jsmeix commented Jul 31, 2017 • edited

schabrolles commented Jul 31, 2017

jsmeix left a comment

Choose a reason for hiding this comment

jsmeix commented Jul 31, 2017

schabrolles commented Jul 28, 2017 •

edited

jsmeix commented Jul 31, 2017 •

edited

jsmeix commented Jul 31, 2017 •

edited