Dry-Run Backup

[KeithScheiwiller]

Related to #10

It would be nice to be able to see what all will be changed via a snapshot before actually creating one. In essence, a way to show a diff between the files stored in restic, and a given path on the local filesystem. This would especially help when making changes to backup configurations so as to be able to catch anything that might be missing/errantly included before hand.

I assume given the architecture, this would be impossible to do without going through the steps of an actual backup. If so, I think it makes more sense to position it as a dry-run backup (where nothing is actually committed) rather than a diff a la git status.

[fd0]

Hm, interesting idea, thanks. Maybe we can add this once #1494 is done, e.g. add some kind of backup --dry-run which only shows what would happen. This will also help developing and testing exclude filters.

[binnisb]

@fd0 Testing exclude filters is the reason I found this issue so 👍 from me.

[Liganic]

looking forward to this, now that #1494 is in, even if it's just super simple first iteration.

[hmage]

Chiming in to say that I want this to test exclude filters before I commit.

An output that shows per-file or per-folder size during dry run would be useful as well to find big culprits that don't need backing up.

[chrissv]

Another vote +1 for this useful feature!

[bjoe2k4]

Also +1 from me, would be extremely helpful!

[EmperorArthur]

Me too. It's not fun having to constantly delete snapshots or terminate a run early and have to clean things up while testing excludes.

[mholt]

I've got my hands full at the moment, but would any of you like to submit a pull request?

[abathur]

@rmmh Thanks for working on this feature!

Mostly commenting to throw more keywords into this issue for everything I tried searching since it took me a good bit to find this specific issue. I am sketching out a pre-flight ready-to-wipe script that needs to see if there is anything for restic to back up in order to check / confirm / test that my working copy and restic are in sync.

I was afraid I'd have to build something bespoke with find, mtime, includes, excludes, and just hope the behavior actually matched.

[jbweston]

Is there any progress on this? I see that @rmmh has implemented this on his fork, but don't see a PR.

[bjoe2k4]

How about #2308

[jbweston]

How about #2308

I didn't see that earlier. In any case it seems that there has been no movement on this since august.

[rawtaz]

From what I can tell the main use case and what pretty much 99% of every request is asking for is the ability to see what files would be backed up, usually to test exclude filters, rather than seeing how much data would be transferred in a backup. I'm thinking we should perhaps implement that, i.e. just scanning and building a list of what would be backed up, rather than a more complicated and heavy process where we also determine how much data would have to be sent. Would that be fine?

[abathur]

@rawtaz IIRC my use-case when I subscribed to this was testing exclude filters. I don't expect special treatment or herculean effort, but I wanted to comment since I did find an additional potential use-case in the meantime: imperatively confirming each backup is up-to-date.

(well-defined narrow backups--not something Sisyphean like a full running system)

I have a large-ish shell script for managing my user state. It has 3 main tasks:

• bootstrap
• backup
• something I unimaginatively call "check" for now, which roughly checks how prepared I am to lose the system or migrate to a new one (and either exit 0 with a visual confirmation of everything checked, or exit 1 with a list of what's wrong and reminders/context for triaging)

It has (something like) this in it:

# - report on restic backup status?
restic:check() { #TODO: unused
	::TODO "restic backup --dry-run (supported after https://github.com/restic/restic/issues/1542; last checked April 8 2020)"
}

routines:check() {
	homedir:repos:check
	package_manager:check
	vpn_configs:check
	os:settings:check
	homedir:untracked:check
	cloud_storage:check
	# restic:check
}

One way to know everything is backed up is to just run the backup routine again--and it's a good answer when I passively run this on a schedule.

But cruft always accumulates. When I'm actively tidying up to move devices, I'll end up running it repeatedly as I iterate on what it finds. I'm not sure what fraction of total backup time --dry-run would save in practice, but I'd see value if it's substantive (especially since not taking a new backup also saves pruning/checking).

[asteppke]

@rawtaz
While a good use case for --dry-run is testing exclude filters another case is the verification of an existing backup. That is if I run a backup and afterwards something along the lines of restic backup --dry-run --read-data shows no changes that would be roughly equivalent to the verify functionality without the requirement of additional disk space.

This would partly address https://forum.restic.net/t/scope-of-restic-check/154, #1280, and https://forum.restic.net/t/what-does-restic-restore-verify-do/1676.

[fmagin]

If you would truly only want to test the exclude files you can easily work around this (on Linux and maybe Mac at least), by using ncdu -X exclude_file /. This will display a file tree of the cumulative folder size minus excluded files which you can then navigate to check if there are (large) files left that you forgot to exclude. Edit exclude_file, rerun the ncdu command and repeat until you are confident in your exclude file.

The issue with this is that it does not know about files already backed up by restic, so this is only really useful for the initial setup and maybe if you combine it with another exclude file generated from the files already backed up by restic. This still won't detect changed files, but this might be a good enough workaround until something is available in restic itself.

[faizshukri]

@rawtaz

In our use case, our customer has storage size limit. so we would need to check first how much data that will actually be transferred before allowing user to proceed with the backup.

[EmperorArthur]

ncdu -X exclude_file /

This either needs to be all over the documentation, or built into the program. I certainly was not aware that ncdu used exactly the same parser as restic. There are so many exclude parsers out there that without explicit statements in the documentation, I can't trust that they will behave the same. Heck, I've certainly seen the documentation make that claim, only for edge cases to crop up.

Another problem with this approach is restic has Windows builds. Heck, I am currently using them in production! External utilities like ncdu are often not available on that environment.

[erincerys]

I submitted a pull request doing what seemed to be left for this from review of #2308. There's test failures though that I don't know how to resolve 😕

[amuckart]

A use-case for --dry-run that isn't mentioned here is building wrapper scripts for restic.

I'm chasing a problem at the moment where the way the script builds the options for restic is creating a timestamp restic is refusing to parse for reasons I can't explain (it works when I copy/paste the generated command line).

The only way to test this is to actually run restic, but without a --dry-run option I'm having to start and abort a backup every time which is painful.

[erincerys]

This code is on master, but hasn't been released yet. The last release was a day before the merge took place... 😭

Can we get this out, pleeeease? @fd0

[dimejo]

@erincerys:

This code is on master, but hasn't been released yet. The last release was a day before the merge took place... sob

Can we get this out, pleeeease? @fd0

Commits to master are automatically tested, compiled and uploaded to beta.restic.net. You can use those releases if you can't build restic yourself.

[darioseidl]

@dimejo Thank you, but I wouldn't want to use a beta release for a production backup. I don't expect that it would break the backups, but I'd rather wait for an official release.

Any idea when the next release will be? The dry-run feature would be really useful to check beforehand how much space a backup will need (to avoid running out of disk space).

[rawtaz]

I use the master build all the time :) If you're worried you could use it only for the dry-run test and then do the real work with the latest release.