HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem! #882

didacog · 2016-06-16T12:36:32Z

Relax-and-Recover (rear) Issue Template

Please fill in the following items before submitting a new issue:

rear version (/usr/sbin/rear -V):
Relax-and-Recover 1.17.2 / Git
OS version (cat /etc/rear/os.conf or lsb_release -a):
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Release: 6.6
Codename: Santiago
rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

    DRLM_NAME=sdomeX01
    OUTPUT=PXE
    OUTPUT_PREFIX=$OUTPUT
    OUTPUT_PREFIX_PXE=$DRLM_NAME/$OUTPUT_PREFIX
    OUTPUT_URL=nfs://192.168.1.100/DRLM/STORE/${DRLM_NAME}
    BACKUP=NETFS
    BACKUP_URL=nfs://192.168.1.100/DRLM/STORE/${DRLM_NAME}
    NETFS_PREFIX=BKP
    EXCLUDE_VG=( 'vgDATA' )
    SSH_ROOT_PASSWORD=rear
    USING_UEFI_BOOTLOADER=1
    AUTOEXCLUDE_MULTIPATH=n
    BOOT_OVER_SAN=y

Brief description of the issue

We are playing with new HP Superdome-X servers. These brand new servers are using multipath an booting from SAN.

On disk layout collection never gets info from root filesystem (/) but for other partitions in same device collects all info with no problems. The problem is that is not backing up de ROOT filesystem!

There is an issue with this file: /usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh

this line:

read_filesystems_command="$findmnt_command **-nrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

# findmnt command looks on  /proc/self/mountinfo by default and gets this info of (/):

$ cat /proc/self/mountinfo | grep ext3
21 1 253:12 / / rw,relatime - ext3 /dev/mapper/**360002ac000000000000000040001a624p4** rw,errors=continue,user_xattr,acl,barrier=1,data=ordered
23 21 253:10 / /boot rw,relatime - ext3 /dev/mapper/vvbootp2 rw,errors=continue,user_xattr,acl,barrier=1,data=ordered

In this case is informing the device WWN from SAN instead of the mpath alias name used in all rear scripts.

# mount command uses /etc/mtab by default gets the correct device info:

$ cat /etc/mtab | grep ext3
/dev/mapper/**vvbootp4** / ext3 rw 0 0
/dev/mapper/vvbootp2 /boot ext3 rw 0 0

Work-around, if any

Proposal:

/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh

Change this line:

read_filesystems_command="$findmnt_command **-nrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

to:

read_filesystems_command="$findmnt_command **-mnrv** -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

This will force findmnt command to use /etc/mtab instead of /proc/self/mountinfo and provide the correct info resulting in the correct disklayout file:

fs /dev/mapper/**vvbootp4** / ext3 uuid=2cf46bd1-892b-4154-8f87-c6fc79fcc2a5 label=HANA_ROOT blocksize=4096 reserved_blocks=5% max_mounts=-1 check_interval=0d bytes_per_inode=16381 default_mount_options=user_xattr,acl options=rw

I've tested it and now is getting the correct info, but prior to sending a pull request I would ask if you guys have any concern about this modification.

Kind Regards!

Didac

The text was updated successfully, but these errors were encountered:

jsmeix · 2016-06-16T12:50:13Z

The code that uses the findmnt command is from me
therefore I assign the issue to me.

I do not run RHEL (and I do not know what is special on RHEL)
therefore I assign the issue also to @gdha

@didacog
simply do a GitHub Pull Request (regardless how good your code
currently is) so that we have something real to discuss it
and so that I can test if it still works on SLES with your changes.
Please use the current rear GitHub master code
as base for your Pull Request.

@didacog
the code snippets in your comment avobe are very hard to read
because of the automated formatting in GitHub.
For code snippets I enclose them
in HTML

and

tags which works o.k.
for me but one has to write < as HTML <
Perhaps there is something even better how to
specify preformatted texts in GitHub?

didacog · 2016-06-16T13:01:51Z

Hi @jsmeix @gdha,

Ok I will prepare the pull request. this is the code to change, just one findmnt option:

/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh
Change this line:
read_filesystems_command="$findmnt_command -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"
to:
read_filesystems_command="$findmnt_command -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

I will also check if any modification needed for this other issue:

2016-06-16 13:22:34.103712191 Could not determine start of partition vvbootp1.
2016-06-16 13:22:34.111206812 Could not determine start of partition vvbootp2.
2016-06-16 13:22:34.118799098 Could not determine start of partition vvbootp3.
2016-06-16 13:22:34.126096733 Could not determine start of partition vvbootp4.

resulting in /var/lib/rear/layout/disklayout.conf:

part /dev/mapper/vvboot 536870912 unknown rear-noname boot /dev/mapper/vvbootp1
part /dev/mapper/vvboot 536870912 unknown rear-noname none /dev/mapper/vvbootp2
part /dev/mapper/vvboot 2147483648 unknown rear-noname none /dev/mapper/vvbootp3
part /dev/mapper/vvboot 271623127040 unknown rear-noname none /dev/mapper/vvbootp4

And provide a pull request with all modifications needed.

Thanks!

Didac

jsmeix · 2016-06-16T13:03:00Z

I changed the initial comment to
use HTML

and

tags
and now I can better read it.

On very first glance I do not like to

force findmnt command to use /etc/mtab instead of /proc/self/mountinfo

because think using /etc/mtab is an indirection (cf. RFC1925 6a)
that can cause whatever unexpected failures instead of
asking the kernel directly (via /proc/self/mountinfo).

@didacog
nevertheless do your Pull Request.

jsmeix · 2016-06-16T13:17:28Z

On my SLES12-SP1 system with its default btrfs structure
the output of the proposed new "findmnt -mnrv ..." is the same
as the output of the currently used "findmnt -nrv ..." so that
from my current point of view the proposed change seems
to be o.k. (i.e. it seems to cause no regression for SLE).

What I did:

# cat /etc/os-release 
NAME="SLES"
VERSION="12-SP1"
VERSION_ID="12.1"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP1"
...
# /usr/bin/findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs >/tmp/findmnt-mnrv
# /usr/bin/findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs >/tmp/findmnt-nrv
# diff -s /tmp/findmnt-nrv /tmp/findmnt-mnrv
Files /tmp/findmnt-nrv and /tmp/findmnt-mnrv are identical

jsmeix · 2016-06-16T13:24:30Z

Nevertheless I need to inspect what the actual difference is
between /etc/mtab which is a link to /proc/self/mounts
versus /proc/self/mountinfo

didacog · 2016-06-16T13:25:50Z

@jsmeix you are really fast! ;)

I think this will be the same behaviour as using mount command if findmnt is not present, and should work.

I'm now deeping with the other issue with partition layouts and after that I will provide a pull request.

Thanks!

didacog · 2016-06-16T13:30:29Z

@jsmeix i found this post regarding /proc/self/mounts vs /proc/self/mountinfo:

http://serverfault.com/questions/581178/proc-self-mountinfo-per-mount-options-vs-per-super-options/581180

jsmeix · 2016-06-16T13:33:02Z

If it is really the exact same behaviour as the old mount command
I could not accept it because I need new findmnt features to deal
correctly with btrfs subvolumes.

See also my comment regarding "man 8 mount"
in 23_filesystem_layout.sh but this comment only
tells to use findmnt instead of mount so that
"findmnt -m..." could be the right thing.

On the other hand perhaps using "findmnt -m..."
might be not an actually right future-proof solution
but merely some kind of workaround that only hides
to deal correctly with the actual root cause?

jsmeix · 2016-06-16T13:40:23Z

@didacog
when you are currently "playing with new HP Superdome-X servers"
could you also test if your changes also work with SLE12-SP1
and its default btrfs subvolume structure?

Obviously I cannot do such a test myself because I do not
have a HP Superdome-X server to play around ;-)

Such a test would help a lot to ensure that also btrfs subvolumes
on RHEL still work on that new hardware or to reveal the
actual root cause (e.g. something like better general support
when the root filesystem is on SAN via multipath.

I never tested rear myself with SAN and multipath.

I blindly guess that general support when the root filesystem
is on SAN via multipath might be currently poor in rear.

jsmeix · 2016-06-16T13:49:16Z

@didacog
many thanks for the link in
#882 (comment)

I do not yet fully understand what is meant in
http://serverfault.com/questions/581178/proc-self-mountinfo-per-mount-options-vs-per-super-options/581180
with (excerpt)

/proc/self/mountinfo is the most authoritative source to check your mounts.
/proc/mounts is a deprecated source to check the status of your mounts.
...
 /proc/mounts symlinks to /proc/self/mounts

As far as I understand it this means that
/proc/self/mounts (and its symlinks like /proc/mounts or /etc/mtab)
are deprecated sources to check the status of your mounts.

Accordingly it seems "findmnt -m..."
is not an actually right future-proof solution.

didacog · 2016-06-16T13:53:11Z

@jsmeix

I think that with the root partition on mpath device, the problem is when kernel loading the root partiton booting from SAN, this partition has not yet the mpath alias defined in kernel and keeps its WWN name on the /proc/self/mountinfo file.

Maybe an exeption when booting from SAN? the problem is that if not properly identified with this name, root filesystem is not present in disklayout file and does not backup of (/).

I will figure out other solutions and use this as workarround in the meantime, and if is possible i will test with SLES, but I cannot promise because those SuperdomeX are not mine :P

didacog · 2016-06-16T14:03:33Z

@jsmeix This case is not with Boot from SAN on SuperdomeX but same output from findmnt command with -m option with btrfs subvolumes on SLES11 SP3:

$ findmnt -nrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t btrfs
/dev/sda4 / btrfs rw,relatime,space_cache
/dev/sda4 /srv btrfs rw,relatime,space_cache
/dev/sda4 /opt btrfs rw,relatime,space_cache
/dev/sda4 /tmp btrfs rw,relatime,space_cache
/dev/sda4 /var/spool btrfs rw,relatime,space_cache
/dev/sda4 /var/run btrfs rw,relatime,space_cache
/dev/sda4 /var/log btrfs rw,relatime,space_cache
/dev/sda4 /var/crash btrfs rw,relatime,space_cache
/dev/sda4 /var/tmp btrfs rw,relatime,space_cache

$ findmnt -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t btrfs
/dev/sda4 / btrfs rw,relatime,space_cache
/dev/sda4 /srv btrfs rw,relatime,space_cache
/dev/sda4 /opt btrfs rw,relatime,space_cache
/dev/sda4 /tmp btrfs rw,relatime,space_cache
/dev/sda4 /var/spool btrfs rw,relatime,space_cache
/dev/sda4 /var/run btrfs rw,relatime,space_cache
/dev/sda4 /var/log btrfs rw,relatime,space_cache
/dev/sda4 /var/crash btrfs rw,relatime,space_cache
/dev/sda4 /var/tmp btrfs rw,relatime,space_cache

Also I've seen that /etc/mtab is a softlink to /proc/self/mounts in SLES but not in RHEL.

SLES:
$ ls -l /etc/mtab
lrwxrwxrwx 1 root root 17 Feb 15 13:47 /etc/mtab -> /proc/self/mounts

RHEL6:
$ ls -l /etc/mtab
-rw-r--r-- 1 root root 990 Jun 16 13:52 /etc/mtab

Maybe we need to use a different approach for each OS.

jsmeix · 2016-06-16T14:20:23Z

In general I am agianst using different approaches
that depend on the Linux distrubution and version
because this is again an indirection (RFC 1925 6a)
that leads to weird unexpected failures because such
code just breaks if on an older Linux distrubution/version
newer software is installed so that the older distrubution
actually behaves like a newer distrubution.

In contrast I prefer to use different approaches
that depend on real existing conditions in the
current system.

I.e. I am against code like

case "$OS_VENDOR" in
    (SUSE_LINUX)
        do_this
        ;;
    (RedHatEnterpriseServer)
        do_that
        ;;
    (*)
        do_default
        ;;
esac

In contrast I prefer code like

if CONDITION ; then
   do_this
else
    do_that
fi

where CONDITION is something real that can be directly
tested on the current system and not something vague
like a version string.

For example I tried to implement my btrfs support code
so that it works generically on SLE and RHEL without
testing things like "$OS_VENDOR" and "$OS_VERSION".

jsmeix · 2016-06-16T14:30:36Z

@didacog
note that in layout/save/GNU/Linux/23_filesystem_layout.sh
I use findmnt at several places:

First to make "fs" entries in disklayout.conf.

Afterwards to make "btrfs..." entries in disklayout.comf.

For the first usage (make "fs" entries) see my comment

# The only difference is that the traditional mount command output has the list of options in parenthesis.

so that here - at last for now - using "findmnt -m"
should be o.k.

In contrast for the other usages (make "btrfs..." entries)
the traditional mount format is insufficient, see my
comment in 23_filesystem_layout.sh and I try to
work aound that by awkward "grepping" in /etc/fstab
which is not a reliably working solution.

Therefore to make make "btrfs..." entries I assume
"findmnt -m" cannot be used - but I did not yet chek it.

didacog · 2016-06-16T15:26:41Z

@jsmeix

Ok, maybe tomorrow I can try to run a backup with this modified on one of the development SLES servers with btrfs subvolumes and check if working ok with findmnt -m.

I guess this should work because, as you can see on the SLES outputs I provided are the same and the origin file format Is very different, but anyway, tomorrow I would clear those doubts :P

jsmeix · 2016-06-17T07:33:21Z

@didacog
very many thanks in advance for your testing.
This is a very valuabe cuntribution to rear!

According to what I wrote above in #882 (comment)
I also think that
at least for now using (deprecated) "findmnt -m"
for making "fs" entries in disklayout.conf should be o.k.
(i.e. adding '-m' for the 'read_filesystems_command')
while in contrast
I think that one cannot use "findmnt -m"
for making "btrfsmountedsubvol" entries in disklayout.conf
(i.e. no '-m' for the 'read_mounted_btrfs_subvolumes_command').

@didacog
because I have basically no knowledge about SAN and multipath
I have a basic question regarding what you wrote in your
initial comment #882 (comment)

device WWN from SAN instead of the mpath alias name

Colud you explain (or provide an URL to explanatory info)
what 'device WWN from SAN' versus 'mpath alias name'
is about?

Via quick "Googling" I found
http://www.datadisk.co.uk/html_docs/redhat/rh_multipathing.htm
but that talks about "WWID".
Is "WWID" the same as "WWN"?

Perhaps the actually right future-proof solution is
to implement in rear a function that gets as input a
'WWID' or 'WWN' and outputs the matching 'mpath alias name'
so that we could still use findmnt without '-m'.

Must each 'WWID' or 'WWN' have exactly one
associated 'mpath alias name' in any case
or could there be none or several 'mpath alias names'
for one 'WWID' or 'WWN'?

Perhaps there is already some ready-to-use command line tool
in the system to find the matching 'mpath alias name'
because I assume when "findmnt -m" outputs
the 'mpath alias name' it may use such kind of mapping
function between WWID or WWN and mpath alias name?

didacog · 2016-06-17T09:27:35Z

@jsmeix

I've tested with SLES and here my results:

...
SLES11 SP3 (rear log):
2016-06-17 10:57:04.940394305 Begin saving filesystem layout
2016-06-17 10:57:04.943144580 Saving filesystem layout (using the findmnt command).
/usr/share/rear/layout/save/GNU/Linux/23_filesystem_layout.sh: line 123: dosfslabel: command not found
findmnt: unknown column: FSROOT
2016-06-17 10:57:05.103735053 End saving filesystem layout
...

I've tested this on shell:

        ########################################
        # Mounted btrfs subvolumes:
        if test -x "$findmnt_command" ; then
            read_mounted_btrfs_subvolumes_command="$findmnt_command -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs"
        else
            read_mounted_btrfs_subvolumes_command="mount -t btrfs | cut -d ' ' -f 1,3,6"
        fi

SLES:
$ findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
findmnt: unknown column: FSROOT
RHEL:
$ findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
findmnt: unknown column: FSROOT

Seems this command is not working at this time.

the resulting disklayout have same output format (because in fact the first findmnt command is getting the fs list and the specific btrfs command is not working)

supported_filesystems="ext2,ext3,ext4,vfat,xfs,reiserfs,btrfs"
read_filesystems_command="$findmnt_command -mnrv -o SOURCE,TARGET,FSTYPE,OPTIONS -t $supported_filesystems"

findmnt -m ... is working also with BTRFS subvols and the dedicated btrfs command is not working.

Look at disklayout outputs:

# grep btrfssnap disklayout.conf (with findmnt -m ... )
# Format: btrfssnapshotsubvol    
#btrfssnapshotsubvol /dev/sda4 / 276 @/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 277 @/opt/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 283 @/.snapshots/4/snapshot
#btrfssnapshotsubvol /dev/sda4 / 284 @/.snapshots/5/snapshot
#btrfssnapshotsubvol /dev/sda4 / 470 @/.snapshots/98/snapshot
#btrfssnapshotsubvol /dev/sda4 / 471 @/opt/.snapshots/96/snapshot
#btrfssnapshotsubvol /dev/sda4 / 608 @/.snapshots/167/snapshot
#btrfssnapshotsubvol /dev/sda4 / 609 @/.snapshots/168/snapshot
#btrfssnapshotsubvol /dev/sda4 / 610 @/.snapshots/169/snapshot
#btrfssnapshotsubvol /dev/sda4 / 611 @/.snapshots/170/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1595 @/.snapshots/662/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1596 @/opt/.snapshots/656/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3036 @/.snapshots/1382/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3037 @/opt/.snapshots/1376/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4261 @/.snapshots/1993/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4262 @/.snapshots/1994/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4263 @/.snapshots/1995/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4265 @/.snapshots/1996/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4532 @/.snapshots/2130/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4533 @/opt/.snapshots/2120/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4820 @/.snapshots/2274/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4821 @/opt/.snapshots/2264/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4868 @/.snapshots/2298/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4869 @/opt/.snapshots/2288/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4916 @/.snapshots/2322/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4917 @/opt/.snapshots/2312/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4964 @/.snapshots/2346/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4965 @/opt/.snapshots/2336/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5012 @/.snapshots/2370/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5013 @/opt/.snapshots/2360/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5060 @/.snapshots/2394/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5061 @/opt/.snapshots/2384/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5108 @/.snapshots/2418/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5109 @/opt/.snapshots/2408/snapshot
...
...
$ grep btrfssnap disklayout.conf.orig (without findmnt -m ... )
# Format: btrfssnapshotsubvol    
#btrfssnapshotsubvol /dev/sda4 / 276 @/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 277 @/opt/.snapshots/1/snapshot
#btrfssnapshotsubvol /dev/sda4 / 283 @/.snapshots/4/snapshot
#btrfssnapshotsubvol /dev/sda4 / 284 @/.snapshots/5/snapshot
#btrfssnapshotsubvol /dev/sda4 / 470 @/.snapshots/98/snapshot
#btrfssnapshotsubvol /dev/sda4 / 471 @/opt/.snapshots/96/snapshot
#btrfssnapshotsubvol /dev/sda4 / 608 @/.snapshots/167/snapshot
#btrfssnapshotsubvol /dev/sda4 / 609 @/.snapshots/168/snapshot
#btrfssnapshotsubvol /dev/sda4 / 610 @/.snapshots/169/snapshot
#btrfssnapshotsubvol /dev/sda4 / 611 @/.snapshots/170/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1595 @/.snapshots/662/snapshot
#btrfssnapshotsubvol /dev/sda4 / 1596 @/opt/.snapshots/656/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3036 @/.snapshots/1382/snapshot
#btrfssnapshotsubvol /dev/sda4 / 3037 @/opt/.snapshots/1376/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4261 @/.snapshots/1993/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4262 @/.snapshots/1994/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4263 @/.snapshots/1995/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4265 @/.snapshots/1996/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4532 @/.snapshots/2130/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4533 @/opt/.snapshots/2120/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4772 @/.snapshots/2250/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4773 @/opt/.snapshots/2240/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4820 @/.snapshots/2274/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4821 @/opt/.snapshots/2264/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4868 @/.snapshots/2298/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4869 @/opt/.snapshots/2288/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4916 @/.snapshots/2322/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4917 @/opt/.snapshots/2312/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4964 @/.snapshots/2346/snapshot
#btrfssnapshotsubvol /dev/sda4 / 4965 @/opt/.snapshots/2336/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5012 @/.snapshots/2370/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5013 @/opt/.snapshots/2360/snapshot
#btrfssnapshotsubvol /dev/sda4 / 5060 @/.snapshots/2394/snapshot
...
...

Regarding to WWN, yes is WWID, the unique name/ID of a Storage LUN. The problem is that after the multipath is loaded to the kernel and applied all aliases, /dev/mapper/WWIDpX no longer exists on system, only the alias /dev/mapper/vvbootp4. This is the cause rear is not able to get (/) information from system getting findmnt info from /proc/self/mountinfo because the block device is trying to get info no longer exists.

jsmeix · 2016-06-17T10:19:10Z

@didacog
many thanks for your testing on SLE11.

You are right.
On SLE11:

# findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT
findmnt: unknown column: FSROOT

The reason why this was not noticed up to now is
that on SLE11 in parctice "nobody" uses btrfs
(and I never tested btrfs on SLE11).

On SLE12 findmnt knows about FSROOT
(and I tested btrfs only on SLE12).

I will provide a fix soon via #883

didacog · 2016-06-17T10:28:14Z

@jsmeix
No problem! we are here to help! ;)

What about -m option? did you test it on SLES12?

On SLES11 with btrfs subvols the output with or without (-m) is the same. Anyway I cannot test it with SLES12 and not sure if something changes with FSROOT option.

jsmeix · 2016-06-17T11:33:25Z

I merged #884
because it still works for me on SLE12-SP1
with its default btrfs structure.

Furthermore with #884
I use 'findmnt -m' to make the 'fs' entries
(but not for other findmnt commands),
cf. #882 (comment)

@didacog
please upgrade to the current rear master to have #884
and do your further work based on that one.

I did not yet test whether or not SLE11 with btrfs
works with #884
because I need a bit of time to prepare a
SLE11 test system with btrfs...

didacog · 2016-06-17T11:43:30Z

Ok, I will test it and provide some feedback ASAP ;)

jsmeix · 2016-06-17T13:13:12Z

A quick test with SLE11-SP4 with btrfs
works for me with #884

During initial installation of SLE11-SP4
I selected in the YaST partitioning dialog
to use the whole harddisk and then
to use btrfs on it.

I got this partitioning:

# parted -l
Model: ATA QEMU HARDDISK (scsi)
Disk /dev/sda: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start   End     Size    Type     File system     Flags
 1      1049kB  165MB   164MB   primary  ext3            boot, type=83
 2      165MB   1735MB  1571MB  primary  linux-swap(v1)  type=82
 3      1735MB  21.5GB  19.7GB  primary  btrfs           type=83

and that btrfs subvolumes:

# btrfs subvolume list -a /        
ID 256 gen 194 top level 5 path /@
ID 258 gen 195 top level 256 path @/tmp
ID 259 gen 166 top level 256 path @/opt
ID 260 gen 166 top level 256 path @/srv
ID 261 gen 126 top level 256 path @/var/crash
ID 262 gen 209 top level 256 path @/var/spool
ID 263 gen 208 top level 256 path @/var/log
ID 264 gen 208 top level 256 path @/var/run
ID 265 gen 166 top level 256 path @/var/tmp
ID 268 gen 195 top level 256 path @/.snapshots

I used this /etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://10.160.4.244/nfs
NETFS_KEEP_OLD_BACKUP_COPY=yes
BACKUP_PROG_INCLUDE=( '/var/tmp/*' '/var/run/*' '/var/spool/*' '/var/log/*' '/var/crash/*' '/tmp/*' '/srv/*' '/opt/*' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
KEEP_BUILD_DIR=""

I did not fully verify it all is really correct after "rear recover"
but the recovered system works and looks o.k.

jsmeix · 2016-06-17T13:52:16Z

@didacog
regarding your #882 (comment)
"I cannot test it with SLES12"

I wonder why you cannot test it with SLES12?

We (i.e. SUSE) have SLE12 installation images available
for free download at
https://www.suse.com/download-linux

Furthermore there is the new public SUSE Beta Program, see
https://www.suse.com/support/beta-program

didacog · 2016-06-17T14:32:07Z

@jsmeix

Tested on SLES12SP1. ;)

findmnt command shows same output with or without (-m) option. In this case, maybe using (-mnrv) on all cases could be a good solution.

here the evidence:

linux-hxww:~ # lsb_release -a
LSB Version:    n/a
Distributor ID: SUSE LINUX
Description:    SUSE Linux Enterprise Server 12 SP1
Release:    12.1
Codename:   n/a
linux-hxww:~ # findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
/dev/sda2 / rw,relatime,space_cache,subvolid=257,subvol=/@ /@
/dev/sda2 /var/tmp rw,relatime,space_cache,subvolid=275,subvol=/@/var/tmp /@/var/tmp
/dev/sda2 /var/lib/mariadb rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mariadb /@/var/lib/mariadb
/dev/sda2 /var/spool rw,relatime,space_cache,subvolid=274,subvol=/@/var/spool /@/var/spool
/dev/sda2 /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images /@/var/lib/libvirt/images
/dev/sda2 /var/opt rw,relatime,space_cache,subvolid=273,subvol=/@/var/opt /@/var/opt
/dev/sda2 /var/log rw,relatime,space_cache,subvolid=272,subvol=/@/var/log /@/var/log
/dev/sda2 /var/lib/pgsql rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/pgsql /@/var/lib/pgsql
/dev/sda2 /var/lib/named rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/named /@/var/lib/named
/dev/sda2 /var/lib/mysql rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mysql /@/var/lib/mysql
/dev/sda2 /var/lib/mailman rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/mailman /@/var/lib/mailman
/dev/sda2 /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash /@/var/crash
/dev/sda2 /usr/local rw,relatime,space_cache,subvolid=264,subvol=/@/usr/local /@/usr/local
/dev/sda2 /tmp rw,relatime,space_cache,subvolid=263,subvol=/@/tmp /@/tmp
/dev/sda2 /opt rw,relatime,space_cache,subvolid=261,subvol=/@/opt /@/opt
/dev/sda2 /srv rw,relatime,space_cache,subvolid=262,subvol=/@/srv /@/srv
/dev/sda2 /home rw,relatime,space_cache,subvolid=260,subvol=/@/home /@/home
/dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi /@/boot/grub2/x86_64-efi
/dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc /@/boot/grub2/i386-pc
linux-hxww:~ # findmnt -mnrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs
/dev/sda2 / rw,relatime,space_cache,subvolid=257,subvol=/@ /@
/dev/sda2 /var/tmp rw,relatime,space_cache,subvolid=275,subvol=/@/var/tmp /@/var/tmp
/dev/sda2 /var/lib/mariadb rw,relatime,space_cache,subvolid=268,subvol=/@/var/lib/mariadb /@/var/lib/mariadb
/dev/sda2 /var/spool rw,relatime,space_cache,subvolid=274,subvol=/@/var/spool /@/var/spool
/dev/sda2 /var/lib/libvirt/images rw,relatime,space_cache,subvolid=266,subvol=/@/var/lib/libvirt/images /@/var/lib/libvirt/images
/dev/sda2 /var/opt rw,relatime,space_cache,subvolid=273,subvol=/@/var/opt /@/var/opt
/dev/sda2 /var/log rw,relatime,space_cache,subvolid=272,subvol=/@/var/log /@/var/log
/dev/sda2 /var/lib/pgsql rw,relatime,space_cache,subvolid=271,subvol=/@/var/lib/pgsql /@/var/lib/pgsql
/dev/sda2 /var/lib/named rw,relatime,space_cache,subvolid=270,subvol=/@/var/lib/named /@/var/lib/named
/dev/sda2 /var/lib/mysql rw,relatime,space_cache,subvolid=269,subvol=/@/var/lib/mysql /@/var/lib/mysql
/dev/sda2 /var/lib/mailman rw,relatime,space_cache,subvolid=267,subvol=/@/var/lib/mailman /@/var/lib/mailman
/dev/sda2 /var/crash rw,relatime,space_cache,subvolid=265,subvol=/@/var/crash /@/var/crash
/dev/sda2 /usr/local rw,relatime,space_cache,subvolid=264,subvol=/@/usr/local /@/usr/local
/dev/sda2 /tmp rw,relatime,space_cache,subvolid=263,subvol=/@/tmp /@/tmp
/dev/sda2 /opt rw,relatime,space_cache,subvolid=261,subvol=/@/opt /@/opt
/dev/sda2 /srv rw,relatime,space_cache,subvolid=262,subvol=/@/srv /@/srv
/dev/sda2 /home rw,relatime,space_cache,subvolid=260,subvol=/@/home /@/home
/dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=259,subvol=/@/boot/grub2/x86_64-efi /@/boot/grub2/x86_64-efi
/dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=258,subvol=/@/boot/grub2/i386-pc /@/boot/grub2/i386-pc
linux-hxww:~ # findmnt -nrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs > findmnt.out
linux-hxww:~ # findmnt -mnrv -o SOURCE,TARGET,OPTIONS,FSROOT -t btrfs > findmnt-m.out
linux-hxww:~ # diff findmnt.out findmnt-m.out

didacog · 2016-06-28T13:42:59Z

@jsmeix

Tomorrow I will test to recover on SLES12 with your latest code for the FSROOT option, but anyway the findmnt (-m ) should be on BTRFS section also, because if not, in case of BTRFS with BOOT from SAN it will fail with same issue. With my last test the output is the same on both cases.
If you haven't any complaint about this, i will test your code with (-m) on btrfs section also and send a pull request with this and other issues i found playing with Boot from SAN these last days.

I will open an issue for them also together with the pull request.

Regards,
Didac

jsmeix · 2016-06-28T14:26:56Z

@didacog
since I had written that new famous
"Dirty hacks welcome" section in
https://github.com/rear/rear/wiki/Coding-Style
any properly documented code that intends
to make it work everywhere is welcome.

In this particular case:
As long as "findmnt -m" works everywhere it is o.k.
and as long as it is properly documented why that '-m'
is used one could easily later change it as needed.

didacog · 2016-06-28T14:32:43Z

@jsmeix

Ok, tomorrow I will submit a new pull request with those changes properly documented.

Regards,
Didac

didacog · 2016-07-05T09:14:39Z

Hi @jsmeix,

I've submitted a new pull request #903 solving findmnt problems (findmnt -m).

I've tested it on SLES 12 and worked.

Also meet a problem with chattr missing on rescue image. Should be solved with the pull request.

I had problems with usr/share/rear/finalize/Linux-i386/23_run_efibootmgr.sh. But is solved in 1.18 with this code:

if [[ ${Dev/mapper//} != $Dev ]] ; then # we have 'mapper' in devname
    # we only expect mpath_partX  or mpathpX or mpath-partX
    case $Disk in
        (*p)     Disk=${Disk%p} ;;
        (*-part) Disk=${Disk%-part} ;;
        (*_part) Disk=${Disk%_part} ;;
        (*)      Log "Unsupported kpartx partition delimiter for $Dev"
    esac
fi

Regards,

jsmeix · 2016-07-07T11:48:52Z

Damn!
I made a typo in the git commit message for #903
where I falsely wrote #822
instead of the right issue #882

jsmeix · 2016-07-07T12:01:08Z

@didacog
regarding "chattr missing on rescue image" see
usr/share/rear/conf/examples/SLE12-SP1-btrfs-example.conf
With your #903
this should now "just work" even automatically.
Many thanks for it!

Regarding your "problems with ... 23_run_efibootmgr.sh"
in #882 (comment)

I think it would be better when you submit a new separated
GitHub issue for that to avoid that several actually separated
issues get mixed up in one single big and fat monster issue.

In particular because I am not at all an expert regarding
bootloader issues I would prefer a separated issue.

Or perhaps just do a separated GitHub pull request that
fixes your "problems with ... 23_run_efibootmgr.sh"?

jsmeix · 2016-07-07T12:50:53Z

@didacog
I think I misunderstood what you meant with
your "problems with ... 23_run_efibootmgr.sh"
in #882 (comment)

In current rear master code your mentioned code
is already there so that you had problems
that are already fixed in current rear master.

FYI:

"git blame" shows for 23_run_efibootmgr.sh
that your mentioned code was provided by
Pavol Domin via git commit bf6742b
and sarching for that git commit ID in
"git log" shows that commit is about
"SAN booting related improvements" which leads to
git commit 42be800
that reads "Merge pull request #695 from pavoldomin/sanboot"
so that we finally know it was #695

@didacog
can I close this particular issue?

I.e. is this particular issue fixed for you
with #903
and #695

didacog · 2016-07-07T13:27:11Z

@jsmeix Yes you can close this issue. Thanks! ;)

jsmeix added the bug The code does not do what it is meant to do label Jun 16, 2016

jsmeix added this to the Rear future milestone Jun 16, 2016

jsmeix assigned jsmeix and gdha Jun 16, 2016

jsmeix added enhancement Adaptions and new features and removed bug The code does not do what it is meant to do labels Jun 16, 2016

jsmeix mentioned this issue Jun 17, 2016

On older systems findmnt does not know about FSROOT #883

Closed

jsmeix mentioned this issue Jun 17, 2016

Messages on screen missing in log file #885

Closed

jsmeix mentioned this issue Jun 23, 2016

Use of invalid option "-U" for mkfs during recover on RHEL 5 #890

Closed

jsmeix mentioned this issue Jul 7, 2016

No code has been generated to restore device /dev/md0 (raid) #822

Closed

jsmeix added the waiting for info label Jul 7, 2016

jsmeix added fixed / solved / done and removed waiting for info labels Jul 7, 2016

jsmeix modified the milestones: Rear v1.19, Rear future Jul 7, 2016

jsmeix closed this as completed Jul 7, 2016

jsmeix mentioned this issue Dec 8, 2017

replace all mount list commands with findmnt #1631

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem! #882

HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem! #882

didacog commented Jun 16, 2016 •

edited by gdha

jsmeix commented Jun 16, 2016 •

edited

didacog commented Jun 16, 2016

jsmeix commented Jun 16, 2016 •

edited

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

didacog commented Jun 16, 2016

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

didacog commented Jun 16, 2016 •

edited

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016 •

edited

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016

jsmeix commented Jun 17, 2016 •

edited

didacog commented Jun 17, 2016

jsmeix commented Jun 17, 2016

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016

didacog commented Jun 28, 2016

jsmeix commented Jun 28, 2016

didacog commented Jun 28, 2016

didacog commented Jul 5, 2016

jsmeix commented Jul 7, 2016

jsmeix commented Jul 7, 2016

jsmeix commented Jul 7, 2016

didacog commented Jul 7, 2016

HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem! #882

HP Superdome-X servers using multipath an booting from SAN not backing up de ROOT filesystem! #882

Comments

didacog commented Jun 16, 2016 • edited by gdha

Relax-and-Recover (rear) Issue Template

jsmeix commented Jun 16, 2016 • edited

didacog commented Jun 16, 2016

jsmeix commented Jun 16, 2016 • edited

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

didacog commented Jun 16, 2016

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

didacog commented Jun 16, 2016 • edited

jsmeix commented Jun 16, 2016

jsmeix commented Jun 16, 2016

didacog commented Jun 16, 2016

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016 • edited

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016

jsmeix commented Jun 17, 2016 • edited

didacog commented Jun 17, 2016

jsmeix commented Jun 17, 2016

jsmeix commented Jun 17, 2016

didacog commented Jun 17, 2016

didacog commented Jun 28, 2016

jsmeix commented Jun 28, 2016

didacog commented Jun 28, 2016

didacog commented Jul 5, 2016

jsmeix commented Jul 7, 2016

jsmeix commented Jul 7, 2016

jsmeix commented Jul 7, 2016

didacog commented Jul 7, 2016

didacog commented Jun 16, 2016 •

edited by gdha

jsmeix commented Jun 16, 2016 •

edited

jsmeix commented Jun 16, 2016 •

edited

didacog commented Jun 16, 2016 •

edited

didacog commented Jun 17, 2016 •

edited

jsmeix commented Jun 17, 2016 •

edited