rear showing that it is archiving 400GB and taking 4 hours, but actually size is around 4GB

[dheeraj86mathur]

Relax-and-Recover (rear) Issue Template

Please fill in the following items before submitting a new issue:

• rear version (/usr/sbin/rear -V): Relax-and-Recover 1.18 / Git

• OS version (cat /etc/rear/os.conf or lsb_release -a):

  LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
  Distributor ID: RedHatEnterpriseServer
  Description: Red Hat Enterprise Linux Server release 6.7 (Santiago)
  Release: 6.7
  Codename: Santiago

• rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

OUTPUT=ISO
BACKUP=RSYNC
BACKUP_URL="rsync://root@lon6gtsdvbld01.markit.partners/export/Server-Rear/"
OUTPUT_URL="rsync://root@lon6gtsdvbld01.markit.partners/export/Server-Rear/"
NETFS_KEEP_OLD_BACKUP_COPY=y
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/mnt' '/media' '/var/crash/' '/export/Server-Rear/')

• Brief description of the issue

Rear using rsync is using very long time and actual rear size 4GB, but output is showing it taking archive of 417322 MiB (400GB !!)

Rsync version : rsync.x86_64 0:3.0.6-12.el6

[root@nj4gtsdvbld01 rear]# rear -v mkbackup
Relax-and-Recover 1.18 / Git
Using log file: /var/log/rear/rear-serverA.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-serverA.iso (73M)
Copying resulting files to rsync://root@serverB/export/Server-Rear/ location
Copying resulting files to rsync location
removed /tmp/rear.aFwz9Zbr7OidqsF/outputfs/serverA/.lockfile' removed directory:/tmp/rear.aFwz9Zbr7OidqsF/outputfs/serverA'
Calculating size of /export/Server-Rear
Calculating size of /home
Calculating size of /
Calculating size of /tmp
Calculating size of /var
Calculating size of /boot
Estimated size of local file systems is 9616 MB
Calculating size of serverB:/export/Server-Rear/
Creating rsync archive on 'serverB:/export/Server-Rear/'
Archived 417322 MiB [avg 25072 KiB/sec]OK <========== ?? why is this showing 400GB ?
Number of files: 94776
Number of files transferred: 76297
Total file size: 437602727041 bytes
Total transferred file size: 437582177920 bytes
Literal data: 437588234278 bytes
Matched data: 0 bytes
File list size: 2159369
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 437647667541
Total bytes received: 2075737

sent 437647667541 bytes received 2075737 bytes 25675383.14 bytes/sec
total size is 437602727041 speedup is 1.00 in 17045 seconds.

df -h output on serverA :

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup01-rootvol
7.8G 2.3G 5.1G 32% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 488M 62M 402M 14% /boot
/dev/mapper/VolGroup01-homevol
2.0G 3.2M 1.9G 1% /home
/dev/mapper/VolGroup01-tmpvol
3.9G 8.5M 3.7G 1% /tmp
/dev/mapper/VolGroup01-varvol
9.8G 1.7G 7.6G 18% /var
/dev/mapper/Rear_vg-rear_lv
2.0T 5.1G 1.9T 1% /export/Server-Rear

pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 VolGroup01 lvm2 a-- 29.47g 3.47g
/dev/sdc1 Rear_vg lvm2 a-- 2.00t 0

vgs
VG #PV #LV #SN Attr VSize VFree
Rear_vg 1 1 0 wz--n- 2.00t 0
VolGroup01 1 5 0 wz--n- 29.47g 3.47g

lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
rear_lv Rear_vg -wi-ao---- 2.00t
homevol VolGroup01 -wi-ao---- 2.00g
rootvol VolGroup01 -wi-ao---- 8.00g
swapvol VolGroup01 -wi-ao---- 2.00g
tmpvol VolGroup01 -wi-ao---- 4.00g
varvol VolGroup01 -wi-ao---- 10.00g

• Work-around, if any
  Not found yet.

[gozora]

@dheeraj86mathur
Don't you have some sparse files somewhere in you backup path?

[dheeraj86mathur]

@gozora , this is freshly built server. No data on any mount. Still have excluded /export/Server-rear in local.conf, which is the only on seperate VG Rear_vg. I am just testing rear-rsync setup, as nfs port is blocked in my environment.

[gozora]

Hmm, looks like ReaR calculates archived size by it self:
ProgressInfo "Archived $((size/1024/1024)) MiB [avg $((size/1024/(SECONDS-starttime))) KiB/sec]"
Now I'm not sure it this is not just some "buffer overflow".
@dheeraj86mathur could you provide /var/log/rear/rear-.log from rear -d -D mkbackup ?

[gdha]

@dheeraj86mathur try to correct the line:

BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/mnt' '/media' '/var/crash/' '/export/Server-Rear/')

by

BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/mnt/*' '/media/*' '/var/crash/*' '/export/Server-Rear/*')

then at least the point points will be created

[jsmeix]

I assume it somehow works meanwhile
because "no news is good news".

FYI:
In general regarding including/excluding things for the backup
there are meanwhile in current ReaR 2.0 the new variables
BACKUP_ONLY_INCLUDE and BACKUP_ONLY_EXCLUDE
so that one can specify exactly what will be included and/or
excluded for the backup (see the description in default.conf).