Parent-snapshot detection fails with changing --files-from

[BenWiederhake]

Output of restic version

restic 0.9.4 compiled with go1.11.5 on linux/amd64

(debian testing, from the debian repository)

What should restic do differently? Which functionality do you think we should add?

In short: With an invocation like restic backup --files-from my-files-and-dirs.lst, restic could be more efficient about choosing a parent snapshot.

In long:

Whenever my-files-and-dirs.lst changes, no matter how slightly, restic apparently sees a different set of paths. Only slightly different, but still different. When searching for a parent snapshot during backup, this leads to unexpected behavior:

On the one hand, restic sees that the path set is different from anything seen ever before, and assumes a completely new backup. All data is re-uploaded files are scanned again, even though only changed data is uploaded:

repository 7b6b235d opened successfully, password is correct
Files:        1905 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repo: 44.372 KiB
processed 1905 files, 346.776 MiB in 0:02
snapshot e50ef85d saved

On the other hand, I made only a small change in my-files-and-dirs.lst, so I expected that only the new files need to be uploaded.

I'm new to restic, so maybe I'm using it wrong. However, using tags does not seem to change automatic parent detection, and --parent latest does not seem to be supported. And I don't want to specify --parent 12345678 manually all the time, and would like to avoid fiddling with restic snapshots on my own.

I'm not sure which feature to propose. There are multiple things that might help:

1. Allow --parent latest to use the latest snapshot, no matter what. This would be helpful for people like me, who only have one guest per repository anyway, but might result in other scenarios.
2. Instead of 1., allow --parent latest-sametags to use the latest snapshot of the same tag set. This would avoid potential problems, and still cover most use cases.
3. Automatic parent detection could try to find a close match in the previous few snapshots, and if it finds one, use that. As far as I can see, a false positive cannot have a bad impact, can it?

What are you trying to do?

I'm making snapshots of parts of my home directory, and have restrictions on the target repository size. So I only want to include specific things, like ~/workspace/, ~/bin/, ~/.bashrc, and so on. But not other things, like the gigantic folder of virtual machines, as the restic host has not enough space for that. Obviously, this list is subject to change.

With the current behavior, a simple backup without --parent 5eab0a7 makes the backup run a bit longer than necessary. Deduplication does its job perfectly, and no excess data is stored.

With the proposed behavior, no such delay would happen, or only when detection fails.

Did restic help you or made you happy in any way?

I'm currently using rsnapshot, and it works great for its use case. However, with so many small files, and some large files constantly changing (thunderbird is a strong offender), my home directory falls outside rsnapshot's use case. It seems restic is fast and space-efficient enough to cope with that much better. Hooray!

---

Just for the record: I no longer have this particular issue, since I personally can just avoid it. I don't close the issue because I recognize that other people do run into trouble due to this issue.

[Ian2020]

It is possible to workaround this with the --parent flag on the backup cmd however it does require the kind of fiddling OP wanted to avoid. For anyone who's interested though the following bash uses the most recent snapshot as the parent:

restic backup --parent $(restic  --json snapshots | jq -r 'max_by(.time) | .short_id')

It requires installing jq to work on the json output.

[robvalca]

It is possible to workaround this with the --parent flag on the backup cmd however it does require the kind of fiddling OP wanted to avoid. For anyone who's interested though the following bash uses the most recent snapshot as the parent:

restic backup --parent $(restic  --json snapshots | jq -r 'max_by(.time) | .short_id')

It requires installing jq to work on the json output.

I think that I tried that at some point but it did not help me. I can give it a try with recent restic version. Thanks !

[robvalca]

I tried the workaround, but in my case, it did not help. This is what I did:

• restic version:

restic 0.9.6 (v0.9.6-81-gc83044f-dirty) compiled with go1.11.5 on linux/amd64

• init repo:

restic -r local:/tmp/restic init

• backup something (twice)

restic -r local:/tmp/restic backup /var/log

• Add something in the tree:

touch /var/log/eos/test

• Getting snapshot information:

ID        Time                 Host                 Tags        Paths
---------------------------------------------------------------------------------
132a2a66  2020-05-19 13:35:08  cbox-restic              /var/log
baf13eeb  2020-05-19 13:35:22  cbox-restic             /var/log
---------------------------------------------------------------------------------
2 snapshots

• Create a file /tmp/files with /var/log/eos/test inside
• Backup again using --files-from and --parent

restic -r local:/tmp/restic backup --files-from /tmp/files --parent baf13eeb
...

• Checking again the snapshot information:

ID        Time                 Host                 Tags        Paths
---------------------------------------------------------------------------------
132a2a66  2020-05-19 13:35:08  cbox-restic            /var/log
baf13eeb  2020-05-19 13:35:22  cbox-restic              /var/log
56eb8d97  2020-05-19 13:37:02  cbox-restic              /var/log/eos/test
---------------------------------------------------------------------------------

• Restoring or browse the latest snapshot only shows the added file, and this snapshot is detached from the previous ones

# restic -r local:/tmp/restic restore 56eb8d97 --target=/tmp/
enter password for repository:
repository c75b712f opened successfully, password is correct
found 4 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
restoring <Snapshot 56eb8d97 of [/var/log/eos/test] at 2020-05-19 13:37:02.851574275 
...
# find /tmp/var/
/tmp/var/
/tmp/var/log
/tmp/var/log/eos
/tmp/var/log/eos/test

[Ian2020]

My understanding is that a parent snapshot is only used during the backup operation and it's only purpose is to improve the speed of the backup (by not scanning files whose metadata is unchanged from that recorded in the parent snapshot). Parent/child relationships are not recorded in the repo.

The manual and cmdline help are light on info but I found this thread on the forums and great explanation from fd0:

A parent snapshot in restic terminology is only used during backup. For an initial backup, restic needs to open and read all files that are to be saved, that’s unavoidable. But for a subsequent backup, restic tries to find a parent snapshot and then loads the metadata which was saved for that snapshot to check if files have been modified. Only modified files are read (and deduplicated) again, whereas for unmodified files restic just takes the contents the previous (“parent”) snapshot recorded. That’s much more efficient.

The only evidence of a parent is when backing up and only if restic has found an appropriate parent or you've selected one yourself with the --parent flag. The effects are:

• A faster backup (restic will jump quickly through the files that haven't changed)
• The stats printed at the end (files modified etc) are in relation to the parent. If no parent is found all files/dirs will show as modified I believe.

[jim-collier]

I'm wondering what the canonical fix or workaround (if any) to the title, is? I've read the web of issues that this one spawns or references, and although it's possible the one true answer (if any) is clearly stated, I don't seem able to parse it out!

I've read through these discussions and subsequent links:

• This one
• Support a user-provided value for snapshot paths #2993
• Add an option to restic backup to override the snapshot's "paths" field #2714
• Forum: "What are parent snapshots?"

Mucho thanks.

[BenWiederhake]

My "workaround" is simply that my --files-from simply doesn't change. However, I use --exclude-file, and the pointed-to file changes somewhat frequently, without causing parent detection to fail.

[MichaelEischer]

I remember seeing some workaround which involved grouping the output of snapshots by tags and hostname to find a snapshot which can be used as the latest one and the explicitly passing it as backup --parent <id>.

[jim-collier]

Well shoot. I'm almost done with a program to generate an exact list of files to back up, easily filtered with regex, date, and size. (And fine-tunable / debuggable with additional logged output of excluded files.) The idea - as is fairly common with largish batch-oriented file ultities - is to run it first to generate a fresh list immediately before every backup run, then pass it to restic with --files-from.

But obviously not if it's going to have to rescan every file every time. (For me that's a bigger problem even if it doesn't upload it again.)

I'd like to open a focused issue to perhaps someday address this, but with so many related issues and discussions - and with only a vague understanding of how snapshot naming and referencing works - I'm not sure how to proceed. Hesitant to just open an ill-informed issue, hope for the best, and potentially add to issue pollution. Any advice on that?

I guess the core of the issue would be something like, "When --files-from is used, default to comparing to the previous snapshot, even if the contents of --files-from changes". Think that would fly?

[jim-collier]

@BenWiederhake - Thanks. I'm curious about this:

My "workaround" is simply that my --files-from simply doesn't change. However, I use --exclude-file, and the pointed-to file changes somewhat frequently, without causing parent detection to fail.

Specifically, why define a general, unchanging set of stuff to back up in --files-from <file>, instead of explicitly as arguments, or via --include-from <file> which seems like that might be more appropriate to what sounds like your particular use case?

[MichaelEischer]

I guess the core of the issue would be something like, "When --files-from is used, default to comparing to the previous snapshot, even if the contents of --files-from changes". Think that would fly?

That merely asks to make the suggested --parent latest option the default when using --files-from. And that's more a suggestion on how to make the solution suggested in this issue description more convenient to use, than an issue on it's own. It might even be a better idea to always fallback to the latest snapshot (maybe matching on tags) for the current host, as a false positive match can't cause much harm. The worst case is that restic uselessly loads some parts of that snapshot, but when the paths don't match then that will stop pretty much immediately.

The backup command has no --include-from <file> option.