11-multiple-backups.adoc

Using Multiple Backups for Relax-and-Recover

Basics

Currently multiple backups are only supported for:

• the internal BACKUP=NETFS method with BACKUP_TYPE=""

• the internal BACKUP=BLOCKCLONE method

• the external BACKUP=BORG method

In general multiple backups are not supported for BACKUP_TYPE=incremental or BACKUP_TYPE=differential because those require special backup archive file names.

The basic idea behind

A "rear mkbackup" run can be split into a "rear mkrescue" run plus a "rear mkbackuponly" run and the result is still the same.

Accordingly "rear mkbackup" can be split into a single "rear mkrescue" plus multiple "rear mkbackuponly" where each particular "rear mkbackuponly" backups only a particular part of the files of the system, for example:

• a backup of the files of the basic system

• a backup of the files in the /home directories

• a backup of the files in the /opt directory

Multiple "rear mkbackuponly" require that each particular "rear mkbackuponly" uses a specific ReaR configuration file that specifies how that particular "rear mkbackuponly" must be done.

Therefore the '-C' command line parameter is needed where an additional ReaR configuration file can be specified.

The basic way how to create multiple backups

Have common settings in /etc/rear/local.conf

For each particular backup specify its parameters in separated additional configuration files like

/etc/rear/basic_system.conf
/etc/rear/home_backup.conf
/etc/rear/opt_backup.conf

First create the ReaR recovery/rescue system ISO image together with a backup of the files of the basic system:

rear -C basic_system mkbackup

Then backup the files in the /home directories:

rear -C home_backup mkbackuponly

Afterwards backup the files in the /opt directory:

rear -C opt_backup mkbackuponly

The basic way how to recover with multiple backups

The basic idea how to recover with multiple backups is to split the "rear recover" into an initial recovery of the basic system followed by several backup restore operations as follows:

Boot the ReaR recovery/rescue system.

In the ReaR recovery/rescue system do the following:

First recover the basic system:

rear -C basic_system recover

Then restore the files in the /home directories:

rear -C home_backup restoreonly

Afterwards restore the files in the /opt directory:

rear -C opt_backup restoreonly

Finally reboot the recreated system.

For more internal details and some background information see #1088

Relax-and-Recover Setup for Multiple Backups

Assume for example multiple backups should be done using the NETFS backup method with 'tar' as backup program to get separated backups for:

• the files of the basic system

• the files in the /home directories

• the files in the /opt directory

Those four configuration files could be used:

/etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup

/etc/rear/basic_system.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

/etc/rear/home_backup.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

/etc/rear/opt_backup.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

The BACKUP_ONLY_INCLUDE setting is described in conf/default.conf.

With those config files creating the ReaR recovery/rescue system ISO image and subsequently backup the files of the system could be done like:

rear mkrescue
rear -C basic_system mkbackuponly
rear -C home_backup mkbackuponly
rear -C opt_backup mkbackuponly

Recovery of that system could be done by calling in the ReaR recovery/rescue system:

rear -C basic_system recover
rear -C home_backup restoreonly
rear -C opt_backup restoreonly

Note that system recovery with multiple backups requires that first and foremost the basic system is recovered where all files must be restored that are needed to install the bootloader and to boot the basic system into a normal usable state.

Nowadays systemd usually needs files in the /usr directory so that in practice in particular all files in the /usr directory must be restored during the initial basic system recovery plus whatever else is needed to boot and run the basic system.

Multiple backups cannot be used to split the files of the basic system into several backups. The files of the basic system must be in one single backup and that backup must be restored during the initial recovery of the basic system.

Relax-and-Recover Setup for Different Backup Methods

Because multiple backups are used via separated additional configuration files, different backup methods can be used.

Assume for example multiple backups should be used to get separated backups for the files of the basic system using the NETFS backup method with 'tar' as backup program and to backup the files in the /home directory using the BORG backup method.

The configuration files could be like the following:

/etc/rear/local.conf

OUTPUT=ISO
REQUIRED_PROGS+=( borg locale )
COPY_AS_IS+=( "/borg/keys" )

/etc/rear/basic_system.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup

/etc/rear/home_backup.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP=BORG
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BORGBACKUP_ARCHIVE_PREFIX="rear"
BORGBACKUP_HOST="borg.server.name"
BORGBACKUP_USERNAME="borg_server_username"
BORGBACKUP_REPO="/path/to/borg/repository/on/borg/server"
BORGBACKUP_PRUNE_KEEP_HOURLY=5
BORGBACKUP_PRUNE_KEEP_WEEKLY=2
BORGBACKUP_COMPRESSION="zlib,9"
BORGBACKUP_ENC_TYPE="keyfile"
export BORG_KEYS_DIR="/borg/keys"
export BORG_CACHE_DIR="/borg/cache"
export BORG_PASSPHRASE='a1b2c3_d4e5f6'
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
export BORG_REMOTE_PATH="/usr/local/bin/borg"

Using different backup methods requires to get all the binaries and all other needed files of all used backup methods into the ReaR recovery/rescue system during "rear mkbackup/mkrescue".

Those binaries and other needed files must be manually specified via REQUIRED_PROGS and COPY_AS_IS in /etc/rear/local.conf (regarding REQUIRED_PROGS and COPY_AS_IS see conf/default.conf).

With those config files creating the ReaR recovery/rescue system ISO image together with a 'tar' backup of the files of the basic system and a separated Borg backup of the files in /home could be done like:

rear -C home_backup mkbackuponly
rear -C basic_system mkbackup

In contrast to the other examples above the Borg backup is run first because Borg creates encryption keys during repository initialization. This ensures the right /borg/keys is created before it will be copied into the ReaR recovery/rescue system by the subsequent "rear mkbackup/mkrescue". Alternatively the ReaR recovery/rescue system could be created again after the Borg backup is done like:

rear -C basic_system mkbackup
rear -C home_backup mkbackuponly
rear -C basic_system mkrescue

Recovery of that system could be done by calling in the ReaR recovery/rescue system:

rear -C basic_system recover
rear -C home_backup restoreonly

Running Multiple Backups and Restores in Parallel

When the files in multiple backups are separated from each other it should work to run multiple backups or multiple restores in parallel.

Whether or not that actually works in your particular case depends on how you made the backups in your particular case.

For sufficiently well separated backups it should work to run multiple different

rear -C backup_config mkbackuponly

or multiple different

rear -C backup_config restoreonly

in parallel.

Running in parallel is only supported for mkbackuponly and restoreonly.

For example like

rear -C backup1 mkbackuponly & rear -C backup2 mkbackuponly & wait

or

rear -C backup1 restoreonly & rear -C backup2 restoreonly & wait

ReaR’s default logging is not prepared for multiple simultaneous runs and also ReaR’s current progress subsystem is not prepared for that. On the terminal the messages from different simultaneous runs are indistinguishable and the current progress subsystem additionally outputs subsequent messages on one same line which results illegible and meaningless output on the terminal.

Therefore additional parameters must be set to make ReaR’s messages and the progress subsystem output appropriate for parallel runs.

Simultaneously running ReaR workflows require unique messages and unique logfile names.

Therefore the PID ('$$') is specified to be used as message prefix for all ReaR messages and it is also added to the LOGFILE value.

The parameters MESSAGE_PREFIX PROGRESS_MODE and PROGRESS_WAIT_SECONDS are described in conf/default.conf.

For example a setup for parallel runs of mkbackuponly and restoreonly could look like the following:

/etc/rear/local.conf

OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup
MESSAGE_PREFIX="$$: "
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"

/etc/rear/basic_system.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

/etc/rear/home_backup.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

/etc/rear/opt_backup.conf

this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"

With those config files creating the ReaR recovery/rescue system ISO image together with a backup of the files of the basic system and then backup the files in /home and /opt in parallel could be done like:

rear -C basic_system mkbackup
rear -C home_backup mkbackuponly & rear -C opt_backup mkbackuponly & wait

Recovery of that system could be done by calling in the ReaR recovery/rescue system:

rear -C basic_system recover
rear -C home_backup restoreonly & rear -C opt_backup restoreonly & wait

Even on a relatively small system with a single CPU running multiple backups and restores in parallel can be somewhat faster compared to sequential processing.

On powerful systems with multiple CPUs, much main memory, fast storage access, and fast access to the backups it is in practice mandatory to split a single huge backup of the whole system into separated parts and run at least the restores in parallel to utilize powerful hardware and be as fast as possible in case of emergency and time pressure during a real disaster recovery.

Remember that system recovery with multiple backups requires that first and foremost the basic system is recovered where all files must be restored that are needed to install the bootloader and to boot the basic system into a normal usable state so that 'rear recover' cannot run in parallel with 'rear restoreonly'.