restic reporting 128 TiB when backing up /var/lib/docker/aufs!!!!!

[rgusick]

Everything has gone smoothly setting up this container to backup my Ubuntu system. I've created an exclude file to avoid wasting time with files and directories that I don't need backed up. The problem appeared when processing /var (specifically /var/lib/docker/aufs). restic is reporting that it needs to analyze up 128 TiB of data!!!!!!!

The actual data:

> du -hs  /var/lib/docker/aufs  
19G     /var/lib/docker/aufs
> find  /var/lib/docker/aufs | wc -l 
696063
> du -hs  /volume1/backups/restic-repo
37G     /volume1/backups/restic-repo

Needless to say my backup for this one subdirectory (which is already in my EXCLUDE list) has effectively made my backup almost 9000x slower (19 GB vs 128 TiB) than it should be!!!!

Help!

[johansmitsnl]

Can you show me your exclude file or command line?

[rgusick]

cat ~/docker/restic/excludes.txt

/data/bin/* #/data/boot/* /data/dev/* #/data/etc/* #/data/home/* #/data/initrd.img/* #/data/initrd.img.old/* /data/lib/* /data/lib32/* /data/lib64/* /data/lost+found/* /data/media/* /data/mnt/* /data/nxfilter/* #/data/opt/* /data/proc/* #/data/root/* /data/run/* /data/sbin/* #/data/sources/* /data/srv/* /data/sys/* /data/tftpboot/* /data/tmp/* #/data/usr/* #/data/var/* #/data/vmlinuz/* #/data/vmlinuz.old/* /data/volume1/* # # # /data/root/downloads/* /data/var/lib/* # # .git aufs cache *~ .cpan \#*\# build_dir bin sbin lib log tmp *.log *.log.*

…

On Tue, Oct 23, 2018 at 2:58 PM Johan Smits ***@***.***> wrote: Can you show me your exclude file or command line? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHIfgwZ-5FU1JZ3_2cTdzvBQIZsNMUxks5un2a3gaJpZM4X2Cvu> .

[johansmitsnl]

is the exclude file available in the docker container?
Can you post you compose file or command line?

[rgusick]

restic-jsmitsnl: restart: always image: jsmitsnl/docker-restic:latest hostname: YYYYYYYYY domainname: XXXXXXX.net container_name: restic-jsmitsnl volumes: - /etc/localtime:/etc/localtime:ro - /volume1/backups:/volume1/backups - /:/data - ${USERDIR}/docker/shared:/shared - ${USERDIR}/docker/restic:/config environment: - CRON_BACKUP_EXPRESSION="0 2 * * *" - CRON_CLEANUP_EXPRESSION="1 0 * * *" - RESTIC_BACKUP_OPTIONS="--exclude-file=/config/excludes.txt --one-file-system" - RESTIC_REPOSITORY=/volume1/backups/restic-repo - RESTIC_PASSWORD_FILE=/config/restic_password.txt - RESTIC_CLEANUP_KEEP_DAILY=7 - RESTIC_CLEANUP_KEEP_WEEKLY=5 - RESTIC_CLEANUP_KEEP_MONTHLY=12 - RESTIC_CLEANUP_KEEP_YEARLY=75 - RESTIC_CLEANUP_OPTIONS="--prune"

…

On Wed, Oct 24, 2018 at 1:35 AM Johan Smits ***@***.***> wrote: is the exclude file available in the docker container? Can you post you compose file or command line? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHIfnlxU2EzHU-G046ym1AET_Phk-ZSks5un_wvgaJpZM4X2Cvu> .

[johansmitsnl]

Looking at the documentation https://restic.readthedocs.io/en/latest/040_backup.html#including-and-excluding-files should you exclude the directory like this: /data/var/lib/** with to asterisks to match all sub directories to?

In my case I don't backup / and only a the data directory. The host contains only docker and I backup only the docker mounted dirs + configs. In your case looks like ${USERDIR}/docker. And I don't exclude anything.