Use b2_hide_file instead of b2_delete_file_version

[Medo42]

Output of restic version

restic 0.9.2 (v0.9.2-1-g3c8bab2b-dirty) compiled with go1.11 on windows/amd64
(yes, I know, doesn't really matter though ;))

What should restic do differently? Which functionality do you think we should add?

The B2 back-end should use b2_hide_file instead of b2_delete_file_version to remove locks, and possibly for all file removals, or offer an option to enable such behavior.

What are you trying to do?

As many others who also created feature requests for similar suggestions, I want to prevent an attacker who succeeds in breaking into my system from deleting / corrupting my backup. One scenario is a manual attack, but it is also not unthinkable that a crypto trojan will be written to search for credentials of common cloud backup providers on machines that it infects.

This particular suggestion solves the problem in a manner specific to the B2 back-end. It may be superseded by a more general solution but is very simple. First some background though.

1. Versioning
B2 has a concept of file versions. API functions such as b2_list_file_names and b2_download_file_by_name only show you the current contents of directories and files, but previous versions of files as well as the contents of removed files may still be available through the use of b2_list_file_versions and b2_download_file_by_id. To make it clear that the file contents are not actually deleted and are still available through the versions API, the file remove operation is called "b2_hide_file". It is possible to set life-cycle rules to automatically delete these old versions after a given number of days, but any version (old or current) can also be deleted manually through b2_delete_file_version.

2. Permissions
In B2, permissions (capabilities) are set on a per-access-key-basis, and are fairly coarse-grained. Of particular interest here are the capabilities "writeFiles" and "deleteFiles". writeFiles allows the application key to write/overwrite/remove files, but not to delete anything from the version history. In particular, it does not allow b2_delete_file_version - the "deleteFiles" capability is needed for that.

In short, application keys without the deleteFiles capability cannot irrevocably destroy information from the bucket, since they cannot meddle with the version history.

3. Ok, so?
The versioning feature of B2 would allow me to protect my restic backup by only giving the automated backup an application key without the "deleteFiles" capability. However, restic currently uses the b2_delete_file_version call to delete locks, so this currently does not work without errors. This could be fixed though by deleting the locks with b2_hide_file instead - it would keep the lock around in the version history, but that should not really cause much bother since lock files are tiny, and you can set up your bucket to automatically delete hidden files after a while.

Maybe it would even be a good idea to embrace the versioning and replace all calls to b2_delete_file_version with b2_hide_file (from what I understand that would actually be much easier to implement). This way, B2 would keep a version history for all operations, which would even make it possible to use commands like prune and forget with restricted access keys, and to recover some earlier state of the bucket in case the repository gets corrupted.

Did restic help you or made you happy in any way?

Restic gives me peace of mind by safeguarding not just my data, but also that on my parents' PC. When they call about strange noises, weird popups and the thing not booting anymore at all, I can always calmly say: Relax, everything is backed up.

[Medo42]

My solution, did a few tests and it seems to work as intended: https://github.com/Medo42/restic/tree/b2-hide-option

[pascalw]

This would be great indeed! I was trying to do the exact same thing of using a key without deleteFiles with Restic but found it didn't work.

[fd0]

Hey, thanks for this (nicely written) feature request! In the past, we made the trade-off to delete files on B2 instead of just hiding them so users would not be surprised that their repo does not shrink after pruning data (and they keep paying the storage fees).

I'm not super familiar with B2, what's their standard policy for hidden files when a new, unconfigured bucket is created? I'd like to pick a reasonable default which works for most users.

Also I'm wondering: This should also be possible by using the rclone backend: call restic with -o rclone.args="serve restic --stdio --drive-use-trash=false" (the default also includes --b2-hard-delete), that should do what you want.

Is that a valid solution? :)

[tbhi]

Also I'm wondering: This should also be possible by using the rclone backend: call restic with -o rclone.args="serve restic --stdio --drive-use-trash=false" (the default also includes --b2-hard-delete), that should do what you want.

Seems to work. I created an application key with capabilities 'readFiles,writeFiles,listBuckets,listFiles' (seems to be only possible via cli) and restic backup finishes without error.

[Medo42]

I'm not super familiar with B2, what's their standard policy for hidden files when a new, unconfigured bucket is created?

As far as I am aware versioning is always enabled, and old versions are only deleted automatically if you explicitly set a lifecycle rule.

As for the rclone solution - I wasn't aware of the rclone backend yet, so I'll have to investigate. It sounds like it would do exactly what I want though, and perhaps it will also solve #1996 at the same time.

[Medo42]

The rclone solution appears to work. Unfortunately it's not an option for me yet because the bandwidth is just as bad for me as on restic without the fix/workaround for #1996, so rclone might have the same issue. I may try to check if it's really the same and raise it with them as well, but in the meantime I'll have to keep using my own patched restic.

[PriceChild]

Hi @Medo42 did you continue with the rclone way forward?

master...Medo42:b2-hide-option looked quite straight forward but I'm assuming there's a good reason it wasn't proposed as a PR?

I'm looking at the same threat model (preventing deletion/corruption) and am guessing I'm going to be adding rclone into the mix.

[Medo42]

Don't read too much into that, and feel free to make it a PR if you think it should still be added. I'm using rclone now.

[MichaelEischer]

There is already a similar PR #2398.

[PriceChild]

@fd0 / @tbhi FYI I believe --drive-use-trash=false is a Google Drive backend option only, it wouldn't affect behaviour when using B2?

Happy to be corrected.

[swazrgb]

As a workaround, I am using the rclone b2 backend. By default, restic also passes --b2-hard-delete, so explicitly set rclone.args to avoid this:

restic -o rclone.args="serve restic --stdio" -r rclone:b2:bucket backup ...

This lets you use an application key which does not have the deleteFiles permission. I am currently using the following set of permissions for my backup app keys:

b2 create-key restic listAllBucketNames,listBuckets,listFiles,readBucketEncryption,readBucketRetentions,readBuckets,readFileLegalHolds,readFileRetentions,readFiles,shareFiles,writeFiles

You could also limit the key to a specific bucket.

It is also important to set up retention rules on the bucket, especially for the locks folder. For example: