Support for Borg #1037
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # This file is part of Relax-and-Recover, licensed under the GNU General | ||
| # Public License. Refer to the included COPYING for full text of license. | ||
| # | ||
| # 10_load_archives.sh | ||
|
|
||
| # Check if BORG_ARCHIVE_PREFIX is correctly set | ||
| # Using '_' could result to some unpleasant site effects, | ||
|
|
||
| # as this character is used as delimiter in latter `for' loop ... | ||
| # Excluding other non aplhanum characeters is not really necessary, | ||
|
|
||
| # however it looks safer to me. | ||
| # I'm sure archive handling can be done better, but no time for it now ... | ||
| if [[ $BORG_ARCHIVE_PREFIX =~ [^a-zA-Z0-9] ]] \ | ||
| || [[ -z $BORG_ARCHIVE_PREFIX ]]; then | ||
| Error "BORG_ARCHIVE_PREFIX must be alphanumeric non emply value only" | ||
|
|
||
| fi | ||
|
|
||
| # Do we have Borg binary? | ||
| has_binary borg | ||
| StopIfError "Could not find Borg binary" | ||
|
|
||
| # Query Borg server for repoitory information and store it to archive_cache. | ||
|
|
||
| # This should avoid repeatingly quering Borg server, which could be slow. | ||
|
|
||
| archive_cache=$TMP_DIR/borg_archive | ||
| borg list $BORG_USERNAME@$BORG_HOST:$BORG_REPO 2> /dev/null > $archive_cache | ||
|
|
||
| # If $rc == 0 we have repository present, but it is empty | ||
| # In my tests, Borg returned RC = 2, when repository was non-existent or | ||
| # due connection problems (bad hostname, etc ...) | ||
| # If repository is present but empty, rc will be set to 0, and initialization | ||
| # will be skipped | ||
|
There was a problem hiding this comment. in general, please globally review this yourself: add a dot at the end of a sentence. |
||
| rc=$? | ||
|
|
||
| # TODO: Add security options for Borg repository initialization ... | ||
| # This might be an Borg connection error, or missing repository | ||
|
|
||
| # If initialization succeedes, we can cast out connection problems | ||
|
There was a problem hiding this comment. never heard "cast out". "rule out" seems more widespread. |
||
| if [ $rc -ne 0 ]; then | ||
| Log "Creating new Borg repository $BORG_REPO on $BORG_HOST" | ||
| borg init -e none $BORG_USERNAME@$BORG_HOST:$BORG_REPO | ||
|
There was a problem hiding this comment. no encryption is a bad default, esp. if hardcoded. open a ticket about this? There was a problem hiding this comment. be careful: env vars starting with BORG_ are used by borg. e.g. borg will read BORG_REPO and expect it to be a full repo URL (or path). There was a problem hiding this comment. @ThomasWaldmann There was a problem hiding this comment. Any variable prefix is o.k. for me for rear Personally I prefer BORGBACKUP_ over BORGB_ |
||
| rc=$? | ||
| fi | ||
|
|
||
| # Borg repository initilization failed in previous step, | ||
|
|
||
| # backup abort is inevitablee | ||
|
|
||
| if [ $rc -ne 0 ]; then | ||
| Error "Could not initialize Borg repository" | ||
| fi | ||
|
|
||
| # Everything should be be OK now, we can extract information about archives | ||
| # Lets find largest suffix in use, and increment it by 1 | ||
|
|
||
| SUFFIX=0 | ||
|
|
||
| for i in \ | ||
| $(cat $archive_cache | grep "^$BORG_ARCHIVE_PREFIX_" | awk '{print $1}'); do | ||
|
There was a problem hiding this comment. btw, "borg list --short" exists. There was a problem hiding this comment. @ThomasWaldmann |
||
| suffix_tmp=$(echo $i | cut -d "_" -f 2) | ||
|
|
||
| if [ $suffix_tmp -gt $SUFFIX ]; then | ||
| SUFFIX=$suffix_tmp | ||
| fi | ||
| done | ||
|
|
||
| SUFFIX=$(($SUFFIX + 1)) | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../NETFS/default/40_create_include_exclude_files.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # This file is part of Relax-and-Recover, licensed under the GNU General | ||
| # Public License. Refer to the included COPYING for full text of license. | ||
| # | ||
| # 20_start_backup.sh | ||
|
|
||
| include_list=() | ||
|
|
||
| # Check if backup-include.txt (created by 40_create_include_exclude_files.sh), | ||
| # really exists | ||
| if [ ! -r $TMP_DIR/backup-include.txt ]; then | ||
| Error "Cant find include list" | ||
|
|
||
| fi | ||
|
|
||
| # Create Borg friendly include list | ||
| for i in $(cat $TMP_DIR/backup-include.txt); do | ||
| include_list+=("$i ") | ||
| done | ||
|
|
||
| Log "Creating archive ${BORG_ARCHIVE_PREFIX}_$SUFFIX \ | ||
| in repository $BORG_REPO on host $BORG_HOST" | ||
|
|
||
| # Start actual Borg backup | ||
| borg create --one-file-system --verbose --stats \ | ||
|
There was a problem hiding this comment. is --one-file-system the usual thing done in REAR? There was a problem hiding this comment. @ThomasWaldmann from what I've understood --one-file-system have similar functionality as -x or --xdev (hence is not crossing mount points). There was a problem hiding this comment. Rear uses '--one-file-system' intentionally for the 'tar' backup |
||
| --compression $BORG_COMPRESSION --exclude-from $TMP_DIR/backup-exclude.txt \ | ||
| $BORG_USERNAME@$BORG_HOST:$BORG_REPO::${BORG_ARCHIVE_PREFIX}_$SUFFIX \ | ||
| ${include_list[@]} | ||
|
|
||
| StopIfError "Failed to create backup" | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # This file is part of Relax-and-Recover, licensed under the GNU General | ||
| # Public License. Refer to the included COPYING for full text of license. | ||
| # | ||
| # 30_prune_old_backups.sh | ||
|
|
||
| prune_opts=() | ||
|
|
||
| # Construct Borg arguments for archive purging | ||
|
|
||
| # No need to check config values of $BORG_PRUNE* family | ||
| # Borg will boil out with error if values are wrong | ||
|
|
||
| if [ ! -z $BORG_PRUNE_HOURLY ]; then | ||
| prune_opts+=("--keep-hourly=$BORG_PRUNE_HOURLY ") | ||
| fi | ||
|
|
||
| if [ ! -z $BORG_PRUNE_DAILY ]; then | ||
| prune_opts+=("--keep-daily=$BORG_PRUNE_DAILY ") | ||
| fi | ||
|
|
||
| if [ ! -z $BORG_PRUNE_WEEKLY ]; then | ||
| prune_opts+=("--keep-weekly=$BORG_PRUNE_WEEKLY ") | ||
| fi | ||
|
|
||
| if [ ! -z $BORG_PRUNE_MONTHLY ]; then | ||
| prune_opts+=("--keep-monthly=$BORG_PRUNE_MONTHLY ") | ||
| fi | ||
|
|
||
| if [ ! -z $BORG_PRUNE_YEARLY ]; then | ||
| prune_opts+=("--keep-yearly=$BORG_PRUNE_YEARLY ") | ||
| fi | ||
|
|
||
| if [ ! -z $prune_opts ]; then | ||
| # Purge old archives according user settings | ||
| Log "Purging old Borg archives in repository $BORG_REPO" | ||
| borg prune -v --list ${prune_opts[@]} $BORG_USERNAME@$BORG_HOST:$BORG_REPO | ||
|
There was a problem hiding this comment. what you name BORG_REPO is a path btw. |
||
| StopIfError "Failed to purge old backups" | ||
| else | ||
| # Purge is not set | ||
| Log "Purgining of old Borg archives not set, skipping" | ||
|
|
||
| fi | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -621,6 +621,39 @@ BEXTRACT_VOLUME= | |
| # eg. Ultrium-1 or /dev/sda1 | ||
| BEXTRACT_DEVICE= | ||
|
|
||
| ## | ||
| # BACKUP=BORG stuff (https://borgbackup.readthedocs.io) | ||
| ## | ||
| # Copy Borg stuff 1:1 | ||
| COPY_AS_IS_BORG=( ) | ||
| # Binaries Borg needs to operate correctly | ||
| # ReaR includes `borg' and `locale' automatically | ||
| PROGS_BORG=( ) | ||
| # Borg server name | ||
| BORG_HOST= | ||
| # Username for connection to Borg server | ||
| BORG_USERNAME= | ||
| # Path to Borg repository on Borg server | ||
| BORG_REPO= | ||
|
|
||
| # Prefix used by ReaR to name archives | ||
| # WARNING: | ||
| # Do not use '_' in BORG_ARCHIVE_PREFIX as it is used internally by ReaR. | ||
| BORG_ARCHIVE_PREFIX="rear" | ||
| # Compression used by Borg | ||
| # Syntax: <compression_type,level> | ||
| # compression_type: none, lz4, zlib, lzma | ||
| # level: 0-9 | ||
| BORG_COMPRESSION="zlib,9" | ||
|
There was a problem hiding this comment. the reason why we default to no compression is that users have to make an informed choice. There was a problem hiding this comment. @ThomasWaldmann OK, I'll change that. There was a problem hiding this comment. I assume with an empty @gozora There was a problem hiding this comment. @jsmeix will do. |
||
| # Borg retention strategy | ||
| BORG_PRUNE_HOURLY= | ||
| BORG_PRUNE_DAILY= | ||
| BORG_PRUNE_WEEKLY= | ||
| BORG_PRUNE_MONTHLY= | ||
| BORG_PRUNE_YEARLY= | ||
| # SSH keys to use for connection to Borg server | ||
| # NOT IMPLEMENTED YET! | ||
| BORG_SSH_KEYS= | ||
|
|
||
| ## | ||
| # BACKUP=BAREOS stuff (bareos.org) | ||
| ## | ||
|
|
@@ -1130,15 +1163,15 @@ PRE_RECOVERY_SCRIPT= | |
|
|
||
| # PRE/POST Backup scripts will provide the ability to run certain tasks before and after a ReaR backup. | ||
| # for example: | ||
| # If a small database running on local filesystem and dependant on a local service, you will maintain its data consistency. | ||
| # Stopping it before backup and restarting again after. | ||
| # In case of any error during backup, if POST tasks were defined, ReaR will run those POST tasks within ExitTasks Array. | ||
| # This will prevent that the database remain stopped. | ||
|
|
||
| # Call this after Relax-and-Recover finished to do anything in the mkbackup/mkbackuponly workflow. | ||
| POST_BACKUP_SCRIPT= | ||
|
|
||
| # Call this before Relax-and-Recover starts to do anything in the mkbackup/mkbackuponly workflow. | ||
| PRE_BACKUP_SCRIPT= | ||
|
|
||
| # some external backup software give you the opportunity to enter paths to exclude ... | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # This file is part of Relax-and-Recover, licensed under the GNU General | ||
| # Public License. Refer to the included COPYING for full text of license. | ||
| # | ||
| # 10_prep_borg.sh | ||
|
|
||
| # Create our own locales, used only for Borg restore | ||
| mkdir -p $ROOTFS_DIR/usr/lib/locale | ||
| localedef -f UTF-8 -i en_US $ROOTFS_DIR/usr/lib/locale/rear.UTF-8 | ||
| StopIfError "Could not create locales" | ||
|
|
||
| # Activate $COPY_AS_IS_BORG from default.conf | ||
| COPY_AS_IS=( "${COPY_AS_IS[@]}" "${COPY_AS_IS_BORG[@]}" ) | ||
|
|
||
| # Activate $PROGS_BORG from default.conf | ||
| # Avoid user to accidentelly override `borg' and `locale' and exclude them | ||
|
|
||
| # from Relax-and-Recover rescue/recovery system | ||
| PROGS=( "${PROGS[@]}" "${PROGS_BORG[@]}" borg locale ) | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
successfully
and i guess the comma after "are" should not be there?