Using Multiple Backups for Relax-and-Recover
Basics
Currently multiple backups are only supported for:
-
the internal BACKUP=NETFS method with BACKUP_TYPE=""
-
the internal BACKUP=BLOCKCLONE method
-
the external BACKUP=BORG method
In general multiple backups are not supported for BACKUP_TYPE=incremental or BACKUP_TYPE=differential because those require special backup archive file names.
The basic idea behind
A "rear mkbackup" run can be split into a "rear mkrescue" run plus a "rear mkbackuponly" run and the result is still the same.
Accordingly "rear mkbackup" can be split into a single "rear mkrescue" plus multiple "rear mkbackuponly" where each particular "rear mkbackuponly" backups only a particular part of the files of the system, for example:
-
a backup of the files of the basic system
-
a backup of the files in the /home directories
-
a backup of the files in the /opt directory
Multiple "rear mkbackuponly" require that each particular "rear mkbackuponly" uses a specific ReaR configuration file that specifies how that particular "rear mkbackuponly" must be done.
Therefore the '-C' command line parameter is needed where an additional ReaR configuration file can be specified.
The basic way how to create multiple backups
Have common settings in /etc/rear/local.conf
For each particular backup specify its parameters in separated additional configuration files like
/etc/rear/basic_system.conf /etc/rear/home_backup.conf /etc/rear/opt_backup.conf
First create the ReaR recovery/rescue system ISO image together with a backup of the files of the basic system:
rear -C basic_system mkbackup
Then backup the files in the /home directories:
rear -C home_backup mkbackuponly
Afterwards backup the files in the /opt directory:
rear -C opt_backup mkbackuponly
The basic way how to recover with multiple backups
The basic idea how to recover with multiple backups is to split the "rear recover" into an initial recovery of the basic system followed by several backup restore operations as follows:
Boot the ReaR recovery/rescue system.
In the ReaR recovery/rescue system do the following:
First recover the basic system:
rear -C basic_system recover
Then restore the files in the /home directories:
rear -C home_backup restoreonly
Afterwards restore the files in the /opt directory:
rear -C opt_backup restoreonly
Finally reboot the recreated system.
For more internal details and some background information see https://github.com/rear/rear/issues/1088
Relax-and-Recover Setup for Multiple Backups
Assume for example multiple backups should be done using the NETFS backup method with 'tar' as backup program to get separated backups for:
-
the files of the basic system
-
the files in the /home directories
-
the files in the /opt directory
Those four configuration files could be used:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backupthis_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"The BACKUP_ONLY_INCLUDE setting is described in conf/default.conf.
With those config files creating the ReaR recovery/rescue system ISO image and subsequently backup the files of the system could be done like:
rear mkrescue rear -C basic_system mkbackuponly rear -C home_backup mkbackuponly rear -C opt_backup mkbackuponly
Recovery of that system could be done by calling in the ReaR recovery/rescue system:
rear -C basic_system recover rear -C home_backup restoreonly rear -C opt_backup restoreonly
Note that system recovery with multiple backups requires that first and foremost the basic system is recovered where all files must be restored that are needed to install the bootloader and to boot the basic system into a normal usable state.
Nowadays systemd usually needs files in the /usr directory so that in practice in particular all files in the /usr directory must be restored during the initial basic system recovery plus whatever else is needed to boot and run the basic system.
Multiple backups cannot be used to split the files of the basic system into several backups. The files of the basic system must be in one single backup and that backup must be restored during the initial recovery of the basic system.
Relax-and-Recover Setup for Different Backup Methods
Because multiple backups are used via separated additional configuration files, different backup methods can be used.
Assume for example multiple backups should be used to get separated backups for the files of the basic system using the NETFS backup method with 'tar' as backup program and to backup the files in the /home directory using the BORG backup method.
The configuration files could be like the following:
OUTPUT=ISO
REQUIRED_PROGS+=( borg locale )
COPY_AS_IS+=( "/borg/keys" )this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backupthis_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}.log"
BACKUP=BORG
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BORGBACKUP_ARCHIVE_PREFIX="rear"
BORGBACKUP_HOST="borg.server.name"
BORGBACKUP_USERNAME="borg_server_username"
BORGBACKUP_REPO="/path/to/borg/repository/on/borg/server"
BORGBACKUP_PRUNE_KEEP_HOURLY=5
BORGBACKUP_PRUNE_KEEP_WEEKLY=2
BORGBACKUP_COMPRESSION="zlib,9"
BORGBACKUP_ENC_TYPE="keyfile"
export BORG_KEYS_DIR="/borg/keys"
export BORG_CACHE_DIR="/borg/cache"
export BORG_PASSPHRASE='a1b2c3_d4e5f6'
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
export BORG_REMOTE_PATH="/usr/local/bin/borg"Using different backup methods requires to get all the binaries and all other needed files of all used backup methods into the ReaR recovery/rescue system during "rear mkbackup/mkrescue".
Those binaries and other needed files must be manually specified via REQUIRED_PROGS and COPY_AS_IS in /etc/rear/local.conf (regarding REQUIRED_PROGS and COPY_AS_IS see conf/default.conf).
With those config files creating the ReaR recovery/rescue system ISO image together with a 'tar' backup of the files of the basic system and a separated Borg backup of the files in /home could be done like:
rear -C home_backup mkbackuponly rear -C basic_system mkbackup
In contrast to the other examples above the Borg backup is run first because Borg creates encryption keys during repository initialization. This ensures the right /borg/keys is created before it will be copied into the ReaR recovery/rescue system by the subsequent "rear mkbackup/mkrescue". Alternatively the ReaR recovery/rescue system could be created again after the Borg backup is done like:
rear -C basic_system mkbackup rear -C home_backup mkbackuponly rear -C basic_system mkrescue
Recovery of that system could be done by calling in the ReaR recovery/rescue system:
rear -C basic_system recover rear -C home_backup restoreonly
Running Multiple Backups and Restores in Parallel
When the files in multiple backups are separated from each other it should work to run multiple backups or multiple restores in parallel.
Whether or not that actually works in your particular case depends on how you made the backups in your particular case.
For sufficiently well separated backups it should work to run multiple different
rear -C backup_config mkbackuponly
or multiple different
rear -C backup_config restoreonly
in parallel.
Running in parallel is only supported for mkbackuponly and restoreonly.
For example like
rear -C backup1 mkbackuponly & rear -C backup2 mkbackuponly & wait
or
rear -C backup1 restoreonly & rear -C backup2 restoreonly & wait
ReaR’s default logging is not prepared for multiple simultaneous runs and also ReaR’s current progress subsystem is not prepared for that. On the terminal the messages from different simultaneous runs are indistinguishable and the current progress subsystem additionally outputs subsequent messages on one same line which results illegible and meaningless output on the terminal.
Therefore additional parameters must be set to make ReaR’s messages and the progress subsystem output appropriate for parallel runs.
Simultaneously running ReaR workflows require unique messages and unique logfile names.
Therefore the PID ('$$') is specified to be used as message prefix for all ReaR messages and it is also added to the LOGFILE value.
The parameters MESSAGE_PREFIX PROGRESS_MODE and PROGRESS_WAIT_SECONDS are described in conf/default.conf.
For example a setup for parallel runs of mkbackuponly and restoreonly could look like the following:
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://your.NFS.server.IP/path/to/your/rear/backup
MESSAGE_PREFIX="$$: "
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="3"this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_PROG_EXCLUDE+=( '/home/*' '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/home/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"
this_file_name=$( basename ${BASH_SOURCE[0]} )
LOGFILE="$LOG_DIR/rear-$HOSTNAME-$WORKFLOW-${this_file_name%.*}-$$.log"
BACKUP_ONLY_INCLUDE="yes"
BACKUP_PROG_INCLUDE=( '/opt/*' )
BACKUP_PROG_ARCHIVE="backup-${this_file_name%.*}"With those config files creating the ReaR recovery/rescue system ISO image together with a backup of the files of the basic system and then backup the files in /home and /opt in parallel could be done like:
rear -C basic_system mkbackup rear -C home_backup mkbackuponly & rear -C opt_backup mkbackuponly & wait
Recovery of that system could be done by calling in the ReaR recovery/rescue system:
rear -C basic_system recover rear -C home_backup restoreonly & rear -C opt_backup restoreonly & wait
Even on a relatively small system with a single CPU running multiple backups and restores in parallel can be somewhat faster compared to sequential processing.
On powerful systems with multiple CPUs, much main memory, fast storage access, and fast access to the backups it is in practice mandatory to split a single huge backup of the whole system into separated parts and run at least the restores in parallel to utilize powerful hardware and be as fast as possible in case of emergency and time pressure during a real disaster recovery.
Remember that system recovery with multiple backups requires that first and foremost the basic system is recovered where all files must be restored that are needed to install the bootloader and to boot the basic system into a normal usable state so that 'rear recover' cannot run in parallel with 'rear restoreonly'.