Feature request: snapshots diff

[rfc1036]

It would be very useful to be able to list the metadata (not files content) differences between two snapshots, to better understand e.g. what needs to be excluded from backups.

(This is a feature of rsnapshot.)

[qyz]

Yes, that would be marvellous. I am missing a connected feature: Especially the size difference between snaphots would be tremendously helpful for me. First of all to answer the question: "Why is the backup taking hours instead of minutes?" The useful metric here would be to see amount of data, that is in one snapshot but not the other.

I'm trying to backup my family's systems to cloud storage over a DSL 2 MBit upload. And fine-tuning the excludes makes the difference between usable and not usable for me. And without seeing the differences between snapshots this is quite difficult. Using du on fuse-mounted restic backups is too slow to be practical within my setup.

This is the last thing that stops me from replacing from my old setup, which has worked pretty well for me for almost 10 years now, with restic,

[fd0]

I have a little treat for you: PR #1462 (branch add-diff) adds a command diff which allows comparing snapshots:

$ ./restic diff 51 79
+    /restic/foo/file
 T   /restic/foo/xxx

Please have a look, try it, and give us feedback :)

Just in case you're not aware of it: You can easily build restic yourself: Install go (>= 1.8), check out the branch, then run go run build.go and you've got a restic binary in the current directory.

[qyz]

Cool, thanks. But I could not get the build from add-diff to work on my saved snapshots.

root@gitlab:~# which restic       # compiled from master on Nov 24
/usr/local/bin/restic
root@gitlab:~# /usr/local/bin/restic snapshots | tail -5
19383d13  2017-12-01 03:37:15  gitlab                  /
fffdfcb4  2017-12-02 03:37:07  gitlab                  /
ae257515  2017-12-03 03:37:07  gitlab                  /
----------------------------------------------------------------------
10 snapshots

root@gitlab:~# which restic.dev   # compiled from add-diff just now
/usr/local/bin/restic.dev
root@gitlab:~# restic.dev diff fffdfcb4 ae257515
unable to open config file: Stat: The specified key does not exist.
Is there a repository at the following location?
s3:https://s3.amazonaws.com/my-bucket-name
root@gitlab:~# /usr/local/bin/restic.dev snapshots | tail -5
unable to open config file: Stat: The specified key does not exist.
Is there a repository at the following location?
s3:https://s3.amazonaws.com/my-bucket-name

Maybe I have caught a bad version from master a few days ago. I don't speak Go, but the changes in add-diff do not look like they could bust functionality in other commands.

Baby is waking up. Maybe I will find time to look into this tonight and try the 0.8.0 release against my snapshots.

[qyz]

Alright, I was bitten by

commit 262b0cd9d42308446dcd4859282ed468ef667d99
Author: Alexander Neumann <alexander@bumpern.de>
Date:   Mon Nov 20 22:29:15 2017 +0100

    s3: Remove default prefix "/restic"

After used the S3 Management Console to move everything from /restic to root within bucket, the add-diff branch works for me, but needs a lot of time: ~85 minutes in my setup. I must admit that my setup does not exactly have a low latency to the S3 bucket: I am in Central Europe, the bucket is in North Virginia:

root@gitlab:/usr/local/bin# time ./restic.dev diff fffdfcb4 ae257515
password is correct
comparing snapshot fffdfcb4 to ae257515:
[…]
real	85m34.067s
user	1m45.443s
sys	0m25.512s

root@gitlab:/# ping -c 1 s3-1-w.amazonaws.com
PING s3-1-w.amazonaws.com (52.216.1.40) 56(84) bytes of data.
64 bytes from s3-1-w.amazonaws.com (52.216.1.40): icmp_seq=1 ttl=46 time=107 ms
[…]
root@gitlab:/# du -sh . 2>/dev/null
15G	.
root@gitlab:/# find / -xdev | wc -l
149260
root@gitlab:/# crontab -l | grep restic
37 3 * * * bash -c ". ~/restic/setup ; time /usr/local/bin/restic backup / -x"

[fd0]

I'm sorry, you probably ran into this (quoting from CHANGLOG.md):

• The s3 backend used the subdir restic within a bucket if no explicit path
  after the bucket name was specified. Since this version, restic does not use
  this default path any more. If you created a repo on s3 in a bucket without
  specifying a path within the bucket, you need to add /restic at the end of
  the repository specification to access your repo: s3:s3.amazonaws.com/bucket/restic
  Undocumented /restic prefix requirement of S3 top-level repositories #1292
  s3: Document and remove default prefix #1437

You need to add /restic to the end of your repo specification. That is backwards compatible and also works with older versions of restic.

[fd0]

@qyz Did you run restic prune with 0.8.0 or newer yet? Because afterwards the cache (added in 0.8.0) will become fully effective. In addition, we've fixed a bug with the cache just today, and I've only just rebased the add-diff branch to master. The commit in question is ad82781

So, if you retry now (maybe after running restic prune over night) with the add-diff branch or master it should be much faster.

[qyz]

I will try that. Thank you!

[qyz]

Excellent! 16 seconds for the diff! Everything worked out as promised. Thank you very very much!

root@gitlab:/# time restic prune
[…]
real	32m50.825s
user	1m33.538s
sys	0m26.089s
root@gitlab:/# time restic diff fffdfcb4 ae257515
[…]
Files:          27 new,    82 removed,  3506 changed
Dirs:            0 new,     0 removed
Others:          0 new,     0 removed
Data Blobs:   1486 new,  1575 removed
Tree Blobs:   5237 new,  5237 removed
  Added:   395.188 MiB
  Removed: 393.600 MiB

real	0m15.895s
user	0m10.142s
sys	0m0.406s