Incremental Backup does not work #1062
Comments
|
It seems that "BACKUP_TYPE=incremental" stuff This is perhaps a good example why we should try I will have a look what goes on here. But don't expect too much from me right now. @gdha |
|
@jottschi FYI: |
jsmeix
changed the title
|
Regarding # Define BACKUP_TYPE (default empty means full backup) or # incremental (only with BACKUP=NETFS and BACKUP_PROG=tar). I.e. BACKUP_TYPE=incremental is only implemented |
|
I can confirm that BACKUP_TYPE=incremental does not work, On my SLES12-SP2 system with current rear master code: # grep -v ^# etc/rear/local.conf
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://10.160.4.244/nfs
REQUIRED_PROGS=( "${REQUIRED_PROGS[@]}" snapper chattr lsattr )
COPY_AS_IS=( "${COPY_AS_IS[@]}" /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default )
BACKUP_PROG_INCLUDE=( '/var/cache/*' '/var/lib/mailman/*' '/var/tmp/*' '/var/lib/pgsql/*' '/usr/local/*' '/opt/*' '/var/lib/libvirt/images/*' '/boot/grub2/i386/*' '/var/opt/*' '/srv/*' '/boot/grub2/x86_64/*' '/var/lib/mariadb/*' '/var/spool/*' '/var/lib/mysql/*' '/tmp/*' '/home/*' '/var/log/*' '/var/lib/named/*' '/var/lib/machines/*' )
POST_RECOVERY_SCRIPT=( 'if snapper --no-dbus -r $TARGET_FS_ROOT get-config | grep -q "^QGROUP.*[0-9]/[0-9]" ; then snapper --no-dbus -r $TARGET_FS_ROOT set-config QGROUP= ; snapper --no-dbus -r $TARGET_FS_ROOT setup-quota && echo snapper setup-quota done || echo snapper setup-quota failed ; else echo snapper setup-quota not used ; fi' )
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
KEEP_BUILD_DIR=""
BACKUP_TYPE="incremental"
FULLBACKUPDAY="Wed"
First run of "rear mkbackup" results on the NFS server /nfs # ls -lh d108 total 993M -rw------- 1 nobody nogroup 825M Nov 9 11:59 2016-11-09-1157-F.tar.gz -rw------- 1 nobody nogroup 202 Nov 9 11:57 README -rw------- 1 nobody nogroup 262 Nov 9 11:57 VERSION -rw------- 1 nobody nogroup 9.6M Nov 9 11:59 backup.log -rw------- 1 nobody nogroup 150M Nov 9 11:57 rear-d108.iso -rw------- 1 nobody nogroup 8.5M Nov 9 11:57 rear.log and in rear-d108.log (excerpt) + source /root/rear/usr/share/rear/prep/NETFS/default/070_set_backup_archive.sh
+++ url_scheme nfs://10.160.4.244/nfs
+++ local url=nfs://10.160.4.244/nfs
+++ local scheme=nfs
+++ grep -q :
+++ echo nfs
+++ echo nfs
++ local scheme=nfs
++ case "$TAPE_DEVICE:$scheme" in
++ '[' incremental == incremental ']'
+++ ls
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=AUTHORS
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=COPYING
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=Makefile
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=README.adoc
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=doc
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=etc
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=packaging
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=usr
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=var
+++ date +%a
++ '[' Wed = Wed ']'
++ Log 'It is Full-Backup-Day'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 11:57:07.965978168 It is Full-Backup-Day'
2016-11-09 11:57:07.965978168 It is Full-Backup-Day
++ rm -f /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/timestamp.txt
++ '[' -f /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/timestamp.txt ']'
++ '[' '!' -f /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/basebackup.txt ']'
++ rm -f /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/timestamp.txt
++ Log 'Timestamp-Files screwd - Performing Full-Backup'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 11:57:07.968021464 Timestamp-Files screwd - Performing Full-Backup'
2016-11-09 11:57:07.968021464 Timestamp-Files screwd - Performing Full-Backup
++ '[' -f /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/timestamp.txt ']'
+++ date +%Y-%m-%d-%H%M
++ backuparchive=/tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/2016-11-09-1157-F.tar.gz
++ date +%Y-%m-%d
/root/rear/usr/share/rear/prep/NETFS/default/070_set_backup_archive.sh: line 46: /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/timestamp.txt: No such file or directory
+++ date +%Y-%m-%d-%H%M
++ echo 2016-11-09-1157-F.tar.gz
/root/rear/usr/share/rear/prep/NETFS/default/070_set_backup_archive.sh: line 47: /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/basebackup.txt: No such file or directory
+++ date +%Y-%m-%d-%H%M
++ BACKUP_PROG_X_OPTIONS=' -V 2016-11-09-1157-F.tar.gz'
++ Log 'Performing Full-Backup /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/2016-11-09-1157-F.tar.gz'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 11:57:07.972137674 Performing Full-Backup /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/2016-11-09-1157-F.tar.gz'
2016-11-09 11:57:07.972137674 Performing Full-Backup /tmp/rear.nIfGXfAbJDj05Rh/outputfs/d108/2016-11-09-1157-F.tar.gz
++ NETFS_KEEP_OLD_BACKUP_COPY=
+ test 1
+ Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
Some minutes later (on the same day) nfs # ls -lh d108 total 1.8G -rw------- 1 nobody nogroup 825M Nov 9 11:59 2016-11-09-1157-F.tar.gz -rw------- 1 nobody nogroup 825M Nov 9 12:12 2016-11-09-1209-F.tar.gz -rw------- 1 nobody nogroup 202 Nov 9 12:10 README -rw------- 1 nobody nogroup 262 Nov 9 12:10 VERSION -rw------- 1 nobody nogroup 9.6M Nov 9 12:12 backup.log -rw-r--r-- 1 nobody nogroup 25 Nov 9 12:09 basebackup.txt -rw------- 1 nobody nogroup 150M Nov 9 12:10 rear-d108.iso -rw------- 1 nobody nogroup 8.5M Nov 9 12:10 rear.log -rw-r--r-- 1 nobody nogroup 11 Nov 9 12:09 timestamp.txt Now the missing files basebackup.txt and timestamp.txt are there. Therefore the first bug seems to be that those files are But rear-d108.log shows why I got a second full backup + source /root/rear/usr/share/rear/prep/NETFS/default/070_set_backup_archive.sh
+++ url_scheme nfs://10.160.4.244/nfs
+++ local url=nfs://10.160.4.244/nfs
+++ local scheme=nfs
+++ grep -q :
+++ echo nfs
+++ echo nfs
++ local scheme=nfs
++ case "$TAPE_DEVICE:$scheme" in
++ '[' incremental == incremental ']'
+++ ls /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1157-F.tar.gz
++ for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
++ restorearchive=/tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1157-F.tar.gz
+++ date +%a
++ '[' Wed = Wed ']'
++ Log 'It is Full-Backup-Day'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 12:09:35.653538566 It is Full-Backup-Day'
2016-11-09 12:09:35.653538566 It is Full-Backup-Day
++ rm -f /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/timestamp.txt
++ '[' -f /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/timestamp.txt ']'
++ '[' '!' -f /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/basebackup.txt ']'
++ rm -f /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/timestamp.txt
++ Log 'Timestamp-Files screwd - Performing Full-Backup'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 12:09:35.661399879 Timestamp-Files screwd - Performing Full-Backup'
2016-11-09 12:09:35.661399879 Timestamp-Files screwd - Performing Full-Backup
++ '[' -f /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/timestamp.txt ']'
+++ date +%Y-%m-%d-%H%M
++ backuparchive=/tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1209-F.tar.gz
++ date +%Y-%m-%d
+++ date +%Y-%m-%d-%H%M
++ echo 2016-11-09-1209-F.tar.gz
+++ date +%Y-%m-%d-%H%M
++ BACKUP_PROG_X_OPTIONS=' -V 2016-11-09-1209-F.tar.gz'
++ Log 'Performing Full-Backup /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1209-F.tar.gz'
++ test 1 -gt 0
+++ Stamp
+++ date '+%Y-%m-%d %H:%M:%S.%N '
++ echo '2016-11-09 12:09:35.683016386 Performing Full-Backup /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1209-F.tar.gz'
2016-11-09 12:09:35.683016386 Performing Full-Backup /tmp/rear.GGFucs3Uy8VXOTK/outputfs/d108/2016-11-09-1209-F.tar.gz
++ NETFS_KEEP_OLD_BACKUP_COPY=
+ test 1
+ Debug 'Leaving debugscripts mode (back to previous bash flags and options settings).'
It states "Timestamp-Files screwd" but that is not right nfs # cat d108/timestamp.txt 2016-11-09 So the second bug is that BACKUP_TYPE=incremental I will try to clean it up and fix the issues... |
|
I think I found out why during my first "rear mkbackup" run
for i in '$(ls ${BUILD_DIR}/outputfs/${NETFS_PREFIX}/*${BACKUP_PROG_SUFFIX}${BACKUP_PROG_COMPRESS_SUFFIX})'
results unexpected files: # Set some bash options: # With nullglob set when e.g. for foo*bar no file matches are found, then foo*bar is removed # (e.g. "ls foo*bar" becomes plain "ls" without "foo*bar: No such file or directory" error). # The extglob shell option enables several extended pattern matching operators. shopt -s nullglob extglob |
|
I did some fixes and clean up @jottschi -06-0015-F.tar.gz--newer=2016-11-06 -V 2016-11-06-0015-F.tar.gz --newer=2016-11-06 -V 2016-11-06-0015-F.tar.gz In my "rear -d -D mkbackup" and "rear -d -D recover" ++ BACKUP_PROG_X_OPTIONS=' --newer=2016-11-09 -V 2016-11-09-1213-F.tar.gz' |
|
Hallo Johannes Am 11.11.2016 14:12 schrieb "Johannes Meixner" notifications@github.com:
|
|
While working on #1066 The current ReaR code supports only one single backup archive But restore/NETFS/default/400_restore_backup.sh is only Therefore the current incremental backup feature Accordingly my #1066 A proper incremental backup feature implementation And support for multiple backup archives is some first |
|
With Nevertheless the current incremental backup implementation @jottschi In general how to test the current ReaR GitHub master code: Basically "git clone" it into a directory and # git clone https://github.com/rear/rear.git # cd rear # vi etc/rear/local.conf # usr/sbin/rear -d -D mkbackup |
|
@danboid regarding your Please test with current ReaR GitHub master code I never used a Samba share as a ReaR output location |
|
Hi jsmeix! I think I read in one of these bug reports that the old incremental code didn't work if you tried to do an incremental update more than once a day? I presume that should be fixed now? Does using incremental mode now backup to flat, uncompressed files instead of 1/2 tarballs? If I wanted to run incremental backups via cron job, are there any X11 tray applets I could use to show me when rear is doing a backup / incremental sync and hopefully show progress (as a percentage) too? If its using rsync then maybe grsync could be used? |
|
With Because the current incremental backup implementation In general regarding issues with the backup: Relax-and-Recover is neither a backup software nor a The general basic question is if more and more backup related In general see also |
|
Regarding |
|
FYI: Perhaps experienced GitHub users who |
rear Version 1.18 and 1.19
OS: Univention Corporate Server UCS 4.1 (a Debian deriviate)
config like this: https://www.harperink.de/?p=2735
BACKUP=NETFS
OUTPUT=ISO
CDROM_SIZE=20
BACKUP_URL=nfs://xxx.xxx.xxx.xxx/volume2/LinuxDR/rear
ISO_DIR=/mnt/ISO
ISO_PREFIX=”rear-$HOSTNAME”
BACKUP_PROG_EXCLUDE=( ‘/tmp/’ ‘/dev/shm/’ ‘/mnt/’ ‘/media/’ $VAR_DIR/output/* )
BACKUP_SELINUX_DISABLE=1
BACKUP_TYPE=incremental
FULLBACKUPDAY=Fri
ReaR make ONE Friday Full, the Incrementals are emtpy:
backup.log
There is a hickup in the tar command :-(
Ralph
The text was updated successfully, but these errors were encountered: