BTRFS subvolumes content are not backup in SLES12-SP2

[schabrolles]

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• rear version (/usr/sbin/rear -V): Relax-and-Recover 2.1-git
• OS version (cat /etc/rear/os.conf or lsb_release -a):

NAME="SLES"
VERSION="12-SP2"
VERSION_ID="12.2"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP2"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp2"
sles12-sp2-seb:/var/log/rear # 

• rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
BACKUP=NETFS
BACKUP_URL=nfs://10.7.99.25/var/www/html/linux/rear
BACKUP_OPTIONS="nfsvers=4,nolock"

## SLES12

REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )

COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

BACKUP_PROG_INCLUDE= ( '/usr/local/*' '/var/lib/pgsql/*' '/var/spool/*' '/var/opt/*' '/home/*' '/var/cache/*' '/var/lib/machines/*' '/var/lib/libvirt/images/*' '/opt/*' '/var/lib/mailman/*' '/var/tmp/*' '/srv/*' '/var/lib/mariadb/*' '/boot/grub2/powerpc-ieee1275/*' '/tmp/*' '/var/lib/mysql/*' '/var/lib/named/*' '/var/log/*' )

POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

• Are you using legacy BIOS or UEFI boot? : POWER ppc64le - PReP

• Brief description of the issue:

ReaR Backup and Restore runs smoothly without any issue. But files contained in btrfs subvolumes are not backuped up even if we specifed the subvolumes in BACKUP_PROG_INCLUDE variable.

For example, analysing the backuplog generated show only /var/spool directory, but not its content.

• Work-around, if any: -

[jsmeix]

For me with SLES12-SP2 on a x86_64 virtual KVM/QEMU machine
it "just works" with current ReaR GitHub master code.

I use the BACKUP_PROG_INCLUDE values from
conf/examples/SLE12-SP2-btrfs-example.conf

# grep  ^BACKUP_PROG_INCLUDE etc/rear/local.conf | tr ' ' '\n' 

BACKUP_PROG_INCLUDE=(
'/var/cache/*'
'/var/lib/mailman/*'
'/var/tmp/*'
'/var/lib/pgsql/*'
'/usr/local/*'
'/opt/*'
'/var/lib/libvirt/images/*'
'/boot/grub2/i386/*'
'/var/opt/*'
'/srv/*'
'/boot/grub2/x86_64/*'
'/var/lib/mariadb/*'
'/var/spool/*'
'/var/lib/mysql/*'
'/tmp/*'
'/home/*'
'/var/log/*'
'/var/lib/named/*'
'/var/lib/machines/*'
)

The backup.log on my NFS server (for "rear -d -D mkbackup")
contains in particular this 'tar' command call:

# grep '^++ tar --warning' backup.log | fold -s -w70

++ tar --warning=no-xdev --sparse --block-number --totals --verbose 
--no-wildcards-match-slash --one-file-system --ignore-failed-read 
--anchored --gzip -X /tmp/rear.jCyzvWhHsTXhBq4/tmp/backup-exclude.txt 
-C / -c -f - /var/cache/cups /var/cache/fontconfig /var/cache/gio-2.0 
/var/cache/krb5rcache /var/cache/ldconfig /var/cache/man 
/var/cache/multipath /var/cache/zypp 
/var/tmp/systemd-private-4b26b95b255c4141b6d6faed5a1b9d16-spice-vdagen
td.service-ClKFKX /usr/local/bin /usr/local/games /usr/local/include 
/usr/local/lib /usr/local/lib64 /usr/local/man /usr/local/sbin 
/usr/local/share /usr/local/src /srv/ftp /srv/www /var/spool/atjobs 
/var/spool/atspool /var/spool/audit /var/spool/clientmqueue 
/var/spool/cron /var/spool/cups /var/spool/locks /var/spool/lpd 
/var/spool/mail /var/spool/plymouth /var/spool/postfix 
/var/spool/rsyslog /var/spool/uucp /tmp/gpg-ZPKcFo 
/tmp/rear.jCyzvWhHsTXhBq4 
/tmp/systemd-private-4b26b95b255c4141b6d6faed5a1b9d16-spice-vdagentd.s
ervice-o3VEwM /home/johannes /var/log/NetworkManager 
/var/log/Xorg.0.log /var/log/YaST2 /var/log/acpid 
/var/log/alternatives.log /var/log/audit /var/log/boot.log 
/var/log/btmp /var/log/cups /var/log/dump /var/log/faillog 
/var/log/firewall /var/log/krb5 /var/log/lastlog /var/log/mail 
/var/log/mail.err /var/log/mail.info /var/log/mail.warn 
/var/log/messages /var/log/news /var/log/ntp /var/log/pbl.log 
/var/log/samba /var/log/snapper.log /var/log/warn /var/log/wtmp 
/var/log/xdm.errors /var/log/zypp /var/log/zypper.log / 
/root/rear.master/var/log/rear/rear-d50.log

On the system where I run "rear -d -D mkbackup"
(with KEEP_BUILD_DIR="yes") the
/tmp/rear.jCyzvWhHsTXhBq4/tmp/backup-include.txt
file contains

/var/cache/*
/var/lib/mailman/*
/var/tmp/*
/var/lib/pgsql/*
/usr/local/*
/opt/*
/var/lib/libvirt/images/*
/boot/grub2/i386/*
/var/opt/*
/srv/*
/boot/grub2/x86_64/*
/var/lib/mariadb/*
/var/spool/*
/var/lib/mysql/*
/tmp/*
/home/*
/var/log/*
/var/lib/named/*
/var/lib/machines/*
/

i.e. the BACKUP_PROG_INCLUDE array members
plus a '/' at the end and the
/tmp/rear.jCyzvWhHsTXhBq4/tmp/backup-exclude.txt
file only contains

/tmp/*
/dev/shm/*
/root/rear.master/var/lib/rear/output/*

so that my backup.tar.gz in particular contains
only the plain tmp/ directory but not its contents.

On my NFS server I get a backup.tar.gz that contains
e.g. for /var/spool

# tar -tf backup.tar.gz | grep 'var/spool/' | sort        
var/spool/
var/spool/atjobs/
var/spool/atjobs/.SEQ
var/spool/atspool/
var/spool/audit/
var/spool/clientmqueue/
var/spool/cron/
var/spool/cron/lastrun/
var/spool/cron/lastrun/cron.daily
var/spool/cron/lastrun/cron.hourly
var/spool/cron/lastrun/cron.monthly
var/spool/cron/lastrun/cron.weekly
var/spool/cron/tabs/
var/spool/cups/
var/spool/cups/tmp/
var/spool/locks
var/spool/lpd/
var/spool/mail/
var/spool/plymouth/
var/spool/postfix/
var/spool/postfix/active/
var/spool/postfix/bounce/
var/spool/postfix/corrupt/
var/spool/postfix/defer/
var/spool/postfix/deferred/
var/spool/postfix/flush/
var/spool/postfix/hold/
var/spool/postfix/incoming/
var/spool/postfix/maildrop/
var/spool/postfix/pid/
var/spool/postfix/pid/master.pid
var/spool/postfix/private/
var/spool/postfix/public/
var/spool/postfix/public/pickup
var/spool/postfix/public/qmgr
var/spool/postfix/saved/
var/spool/postfix/trace/
var/spool/rsyslog/
var/spool/uucp/
var/spool/uucp/uucp/

and I get all them back on another x86_64 virtual KVM/QEMU machine
where I run "rear recover".

FYI
there are some special files in /var/spool/postfix
on the original system that are not in the backup
because those files are sockets, namely

/var/spool/postfix/private/anvil: socket
/var/spool/postfix/private/bounce: socket
/var/spool/postfix/private/defer: socket
/var/spool/postfix/private/discard: socket
/var/spool/postfix/private/error: socket
/var/spool/postfix/private/lmtp: socket
/var/spool/postfix/private/local: socket
/var/spool/postfix/private/proxymap: socket
/var/spool/postfix/private/proxywrite: socket
/var/spool/postfix/private/relay: socket
/var/spool/postfix/private/retry: socket
/var/spool/postfix/private/rewrite: socket
/var/spool/postfix/private/scache: socket
/var/spool/postfix/private/smtp: socket
/var/spool/postfix/private/trace: socket
/var/spool/postfix/private/verify: socket
/var/spool/postfix/private/virtual: socket
/var/spool/postfix/public/cleanup: socket
/var/spool/postfix/public/flush: socket
/var/spool/postfix/public/showq: socket

[jsmeix]

@schabrolles
do you have your SLES12-SP2 new installed from scratch
or is your SLES12-SP2 a service pack update e.g. from
a SLES12-SP1 or from a SLES12-SP0/GA system?

[schabrolles]

@jsmeix it is a pure SLES12-SP2 from DVD

[schabrolles]

@jsmeix
Ok I got it ... my mistake !! (Shame on Me)

There is a "space" between BACKUP_PROG_INCLUDE= and the first (
it works if I remove it.

[jsmeix]

@schabrolles
thank you so much for your fast "all-clear" reply.

I got a bit nervous what obscure stuff might happen here
that may keep my mind subliminally busy over the weekend.

It seems you are a bit unlucky with "too much spaces", cf.
https://github.com/rear/rear/pull/1356/files
;-)

Enjoy your weekend!

[schabrolles]

@jsmeix :-) Arrrg ... +1 point ...

I've tweaked a little my local.conf to make this BACKUP_PROG_INCLUDE variable more dynamic.
It will avoid a new created btrfs subvolume to be excluded from the backup because the variable was not updated.

## SLES12

REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )

COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )

for subvol in $(findmnt -n -r -t btrfs | cut -d ' ' -f 1 | grep -v '^/$' | egrep -v 'snapshots|crash') ; do
        BACKUP_PROG_INCLUDE=( "${BACKUP_PROG_INCLUDE[@]}" "$subvol" )
done

POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )

but I think it could be better to add this stuff in a script that manage "btrfs backup"...
The best would be to mask this btrfs backup complexity to the user.

[jsmeix]

As soon as I really understand the SUSE btrfs complexity
I could try to remove that complexity from the user, but cf.
#1368 (comment)

Regarding the btrfs backup complexity:
I do not know what btrfs subvolumes should be in the backup.
Just all of them (except snapshots except those snapshot
that is currently mounted at '/')?
Perhaps your proposal how to make BACKUP_PROG_INCLUDE
more dynamic is a good starting point for a reasonable
default behaviour.
In the end the user must still check if the default behaviour
is what he wants and if not he can and must adapt his config.
I only like to avoid that users blindly trust the defaults
and later complain when "rear recover" does not result
what they had expected (but never verified), cf.
https://github.com/rear/rear/wiki/Developers-Guide

Regarding "new created btrfs subvolume":
Careful with that on SLE12!
Regarding how complicate it is to create an additional
btrfs subvolume in compliance with the SLE12
default btrfs subvolume structure, you may have a look
at my (meanwhile probably already somewhat outdated)
selfmade personal quick and dirty script in
https://lists.opensuse.org/opensuse-factory/2016-10/msg00397.html