Add an option to restic backup to override the snapshot's "paths" field

[asayers]

Output of restic version

restic 0.9.6 compiled with go1.13.4 on linux/amd64

What should restic do differently? Which functionality do you think we should add?

Add an option to restic backup to override the snapshot's "paths" field.

What are you trying to do?

My first-time restic experience went like this:

• Looks like it has some features for excluding patterns etc., but I want more control over which files are backed up.
• Ah, no problem, it has an option for that: restic backup --files-from lets me just tell it exactly which files to backup. Great!
• Hmm, it's actually treating the contents not as files, but as patterns. Never mind, I'll just escape any glob characters in the file paths, and...
• Ok, it worked! Uploading the snapshot manifest at the end took a strangely long time, but 🤷. Now let's take a look at restic snapshots, and OOOH sweet jesus!

As you can probably guess, I had created a snapshot with 140k items in its paths field 😬. As far as I can tell, this is not actually causing any real problems. I'd just like to be able to override the "path" field when I make a backup, so that the output of restic snapshots is more usable.

Did restic help you today? Did it make you happy in any way?

Restic seems really good. I'm a long-time bup user, and bup is great too - but I wanted encryption so I could use a cloud storage provider. Restic seems to follow mostly the same design principles as bup, so I thought I'd give it a try. I'm liking it so far!

[robvalca]

We are looking for the same feature! Currently at CERN we are backing up around 170M files everyday with restic. The problem is that sometimes we are traversing a quite big folders just for a few changes that we can get quickly from namespace information prior to the backup. This is causing a lot of activity on our instances and affects user experience. Would be nice to pass directly these files to restic but linked to the main path of the backup. Something like restic backup —files-from file.txt /my/path, assuming that files in file.txt would be in the form /my/path/file and using /my/path internally in the repo so then is linked to previous backups.

And @asayers, is not just a visual thing, because if you restore that snapshot you will only get the files specified in the file, not the incremental view of the path. The same for the fuse view.

[Ian2020]

I would add that it also seems to interfere with restic selecting a parent snapshot at backup as the paths field differs for each snapshot (if you have any new/deleted files). In fact this has been filed as a bug already: #2246 so a solution here could fix two bugs at once.

Perhaps instead of allowing the user to set the paths field restic could set the field to the smallest unique path(s) from all the files that have been added to the snapshot? i.e if the snapshot contains:

/home/a/foo
/home/b/bar
/var/log/a
/var/log/b

It would set paths to /home/, /var/log.

[rawtaz]

Thanks for the suggestions. Marking as feature enhancement for now. There's lots of other issues to look at currently.

[MichaelEischer]

@chrahunt reported in #2993 a few further issues caused by the large number of paths:

These are the problems I have:

1. Every backup, a new 11MB snapshot file is created and uploaded. This results in about 1 GB/day extra storage usage.
2. restic snapshots --compact latest is slow, up to 15 seconds. I have to run this to retrieve the parent snapshot and avoid backup delays, similar to Parent-snapshot detection fails with changing --files-from #2246.
3. When running restic forget, I need to specify --group-by host, since the snapshot paths are not usable to me for grouping.
4. If I want to list snapshots, I must always use --compact, so paths are excluded. I must never use --json, since paths are always included.

If restic accepted an option like I described above, then I can pass --snapshot-path /.

[aawsome]

Something like restic backup —files-from file.txt /my/path, assuming that files in file.txt would be in the form /my/path/file and using /my/path internally in the repo so then is linked to previous backups.

In #3200 I propose a new option --set-path to backup which completely overwrites the paths. IMO this is the most flexible way. In your case, you could then use restic backup --files-from file.txt --set-path /my/path, but also something like restic backup --files-from file.txt /other/path --set-path /my/path --set-path /other/path is working (just make sure you --set-path all paths that are included in the backup).

[asayers]

@aawsome Great, that's exactly what I was looking for!

[remram44]

#1376 is a duplicate of this

I am running into this issue too, I am importing snapshots from another system into Restic, and I can't do that at the original path (which contains current data).

[andrewchen5678]

+1