Verify before restore

[stang]

What is the purpose of this change? What does it change?

• extract the operations done by the --verify option into function of Restorer
• verify the content of a node before restoring it (skip if already matching)

Was the change discussed in an issue or in the forum before?

#2011
Checklist

• I have read the Contribution Guidelines
• I have added tests for all changes in this PR
• I have added documentation for the changes (in the manual)
• There's a new file in changelog/unreleased/ that describes the changes for our users (template here)
• I have run gofmt on the code in all commits
• All commit messages are formatted in the same style as the other commits in the repo
• I'm done, this Pull Request is ready for review

[codecov-io]

Codecov Report

Merging #2012 into master will decrease coverage by 4.22%.
The diff coverage is 21.42%.

@@            Coverage Diff             @@
##           master    #2012      +/-   ##
==========================================
- Coverage   50.12%   45.89%   -4.23%     
==========================================
  Files         169      169              
  Lines       13563    13569       +6     
==========================================
- Hits         6798     6228     -570     
- Misses       5742     6372     +630     
+ Partials     1023      969      -54

Impacted Files	Coverage Δ
internal/restorer/restorer.go	47.79% <21.42%> (+2.4%)	⬆️
internal/backend/b2/b2.go	0% <0%> (-78.95%)	⬇️
internal/backend/swift/swift.go	0% <0%> (-78.45%)	⬇️
internal/backend/gs/gs.go	0% <0%> (-73.61%)	⬇️
internal/backend/azure/azure.go	0% <0%> (-69.46%)	⬇️
internal/backend/swift/config.go	36.95% <0%> (-54.35%)	⬇️
cmd/restic/global.go	28.57% <0%> (+0.85%)	⬆️
internal/backend/s3/s3.go	62.4% <0%> (+2.25%)	⬆️
internal/archiver/blob_saver.go	97.53% <0%> (+2.46%)	⬆️
internal/archiver/file_saver.go	88.59% <0%> (+4.38%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0882aca...3c0bf62. Read the comment docs.

[ifedorenko]

I suggest this wait until after #1719

[fd0]

Thank you very much for the contribution! I think @ifedorenko is right: This needs to wait until #1719 is merged.

[bbashy]

That's been merged now but there's conflicts.

[fd0]

That's expected: #1719 changed the same code. This needs to be resolved, preferably by @stang :)

[ifedorenko]

Few notes/observations

• I suggest to make verify-before-restore behaviour optional. For example, it is likely to slow down restore if the target file system is on 2.5" laptop hdd (i.e. slow mechanical disk) and the backend is on fast 1Gbps connection.
• I think we should investigate if pre-restore file content verification should run on single or multiple threads. This likely depends on the type of disks and filesystem used but I don't know if we should use different concurrency strategy on ssd vs hdd, for example.
• It may be desirable to implement verify-before-restore at blob level, which will allow to resume interrupted restore and incremental restore of large files.
• We need to decide what to do with existing files that are not part of the snapshot, i.e. should restore delete or keep them?

[stang]

Sorry, I was on holiday for a little while.

I'll try to have a look and update this PR, taking @ifedorenko inputs in consideration.

[fd0]

Hey, are you still interested in working on this? Or are you stuck and need help? :)

[stang]

Sorry, TBH, I didn't manage to make much time to work on this since my last comment — we're still running on a forked build that seem to do the job for now, and got other stuff prioritised so I'm not sure when I'll be able to pick it up again.

If someone is keen to take-over, I won't say no. If not, I'll just keep it in my backlog.

[MichaelEischer]

Superseded by #3425.