Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use b2_hide_file instead of b2_delete_file_version #2134

Closed
Medo42 opened this issue Jan 2, 2019 · 11 comments · Fixed by #2398
Closed

Use b2_hide_file instead of b2_delete_file_version #2134

Medo42 opened this issue Jan 2, 2019 · 11 comments · Fixed by #2398
Labels

Comments

@Medo42
Copy link

Medo42 commented Jan 2, 2019

Output of restic version

restic 0.9.2 (v0.9.2-1-g3c8bab2b-dirty) compiled with go1.11 on windows/amd64
(yes, I know, doesn't really matter though ;))

What should restic do differently? Which functionality do you think we should add?

The B2 back-end should use b2_hide_file instead of b2_delete_file_version to remove locks, and possibly for all file removals, or offer an option to enable such behavior.

What are you trying to do?

As many others who also created feature requests for similar suggestions, I want to prevent an attacker who succeeds in breaking into my system from deleting / corrupting my backup. One scenario is a manual attack, but it is also not unthinkable that a crypto trojan will be written to search for credentials of common cloud backup providers on machines that it infects.

This particular suggestion solves the problem in a manner specific to the B2 back-end. It may be superseded by a more general solution but is very simple. First some background though.

1. Versioning
B2 has a concept of file versions. API functions such as b2_list_file_names and b2_download_file_by_name only show you the current contents of directories and files, but previous versions of files as well as the contents of removed files may still be available through the use of b2_list_file_versions and b2_download_file_by_id. To make it clear that the file contents are not actually deleted and are still available through the versions API, the file remove operation is called "b2_hide_file". It is possible to set life-cycle rules to automatically delete these old versions after a given number of days, but any version (old or current) can also be deleted manually through b2_delete_file_version.

2. Permissions
In B2, permissions (capabilities) are set on a per-access-key-basis, and are fairly coarse-grained. Of particular interest here are the capabilities "writeFiles" and "deleteFiles". writeFiles allows the application key to write/overwrite/remove files, but not to delete anything from the version history. In particular, it does not allow b2_delete_file_version - the "deleteFiles" capability is needed for that.

In short, application keys without the deleteFiles capability cannot irrevocably destroy information from the bucket, since they cannot meddle with the version history.

3. Ok, so?
The versioning feature of B2 would allow me to protect my restic backup by only giving the automated backup an application key without the "deleteFiles" capability. However, restic currently uses the b2_delete_file_version call to delete locks, so this currently does not work without errors. This could be fixed though by deleting the locks with b2_hide_file instead - it would keep the lock around in the version history, but that should not really cause much bother since lock files are tiny, and you can set up your bucket to automatically delete hidden files after a while.

Maybe it would even be a good idea to embrace the versioning and replace all calls to b2_delete_file_version with b2_hide_file (from what I understand that would actually be much easier to implement). This way, B2 would keep a version history for all operations, which would even make it possible to use commands like prune and forget with restricted access keys, and to recover some earlier state of the bucket in case the repository gets corrupted.

Did restic help you or made you happy in any way?

Restic gives me peace of mind by safeguarding not just my data, but also that on my parents' PC. When they call about strange noises, weird popups and the thing not booting anymore at all, I can always calmly say: Relax, everything is backed up.

@Medo42
Copy link
Author

Medo42 commented Jan 2, 2019

My solution, did a few tests and it seems to work as intended: https://github.com/Medo42/restic/tree/b2-hide-option

@pascalw
Copy link

pascalw commented Mar 27, 2019

This would be great indeed! I was trying to do the exact same thing of using a key without deleteFiles with Restic but found it didn't work.

@fd0 fd0 added the type: feature suggestion suggesting a new feature label Mar 27, 2019
@fd0
Copy link
Member

fd0 commented Mar 27, 2019

Hey, thanks for this (nicely written) feature request! In the past, we made the trade-off to delete files on B2 instead of just hiding them so users would not be surprised that their repo does not shrink after pruning data (and they keep paying the storage fees).

I'm not super familiar with B2, what's their standard policy for hidden files when a new, unconfigured bucket is created? I'd like to pick a reasonable default which works for most users.

Also I'm wondering: This should also be possible by using the rclone backend: call restic with -o rclone.args="serve restic --stdio --drive-use-trash=false" (the default also includes --b2-hard-delete), that should do what you want.

Is that a valid solution? :)

@tbhi
Copy link

tbhi commented Mar 27, 2019

Also I'm wondering: This should also be possible by using the rclone backend: call restic with -o rclone.args="serve restic --stdio --drive-use-trash=false" (the default also includes --b2-hard-delete), that should do what you want.

Seems to work. I created an application key with capabilities 'readFiles,writeFiles,listBuckets,listFiles' (seems to be only possible via cli) and restic backup finishes without error.

@Medo42
Copy link
Author

Medo42 commented Mar 27, 2019

I'm not super familiar with B2, what's their standard policy for hidden files when a new, unconfigured bucket is created?

As far as I am aware versioning is always enabled, and old versions are only deleted automatically if you explicitly set a lifecycle rule.

As for the rclone solution - I wasn't aware of the rclone backend yet, so I'll have to investigate. It sounds like it would do exactly what I want though, and perhaps it will also solve #1996 at the same time.

@Medo42
Copy link
Author

Medo42 commented Mar 27, 2019

The rclone solution appears to work. Unfortunately it's not an option for me yet because the bandwidth is just as bad for me as on restic without the fix/workaround for #1996, so rclone might have the same issue. I may try to check if it's really the same and raise it with them as well, but in the meantime I'll have to keep using my own patched restic.

@PriceChild
Copy link

Hi @Medo42 did you continue with the rclone way forward?

master...Medo42:b2-hide-option looked quite straight forward but I'm assuming there's a good reason it wasn't proposed as a PR?

I'm looking at the same threat model (preventing deletion/corruption) and am guessing I'm going to be adding rclone into the mix.

@Medo42
Copy link
Author

Medo42 commented Oct 18, 2020

Don't read too much into that, and feel free to make it a PR if you think it should still be added. I'm using rclone now.

@MichaelEischer
Copy link
Member

There is already a similar PR #2398.

@PriceChild
Copy link

@fd0 / @tbhi FYI I believe --drive-use-trash=false is a Google Drive backend option only, it wouldn't affect behaviour when using B2?

Happy to be corrected.

@swazrgb
Copy link

swazrgb commented Mar 13, 2022

As a workaround, I am using the rclone b2 backend. By default, restic also passes --b2-hard-delete, so explicitly set rclone.args to avoid this:

restic -o rclone.args="serve restic --stdio" -r rclone:b2:bucket backup ...

This lets you use an application key which does not have the deleteFiles permission. I am currently using the following set of permissions for my backup app keys:

b2 create-key restic listAllBucketNames,listBuckets,listFiles,readBucketEncryption,readBucketRetentions,readBuckets,readFileLegalHolds,readFileRetentions,readFiles,shareFiles,writeFiles

You could also limit the key to a specific bucket.

It is also important to set up retention rules on the bucket, especially for the locks folder. For example:

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants