"I'm more concerned about something crawling into the machine and finding the credentials to the repository somewhere in Kopia's config and using that to mass delete files." - @Neurrone
A shared document has been created to facilitate a transparent discussion logging design choicese, tweaking to changes in Kopia and for drafting documentation. Feel free support and comment. If you want editing rights to this document, please send me your gmail address on slack.
This issue is based on a conversation on slack that took place on slack from May 8th 2021 onwards.
Related to #711, #743 and #913
User story (B2 specific, but please read the question S3-generic):
As a Kopia user weary of randsomware attacks on backups,
I would like to use S3-compatible object locking
so that files cannot be deleted by randsomware or other malicious actors (that do not have access to my (MFA-protected) S3-storage account).
How would or how could Kopia best relate to a) default B2 bucket retention policies (that lock a file for a set number of days) and b) to lifecycle rules (that allow for deletion after a set number of days to save money on the storage bill)?
The challenge I imagine is in cleaning up unreferenced data at some point. Perhaps a maintainence flag could be set with the amount of days after which the S3-storage provider will delete the file. Repackaging content that is still being referenced in to be deleted files would be trivial I imagine.
## Without using Object Locking / immutability the following should be possible
Note that it should already be possible to use one pair of application keys for backing up and restoring (without delete capability) and another for maintenance (that has delete abilities).
Question by Dickson Tan 7:04 AM May 9 2021
Thanks for confirming my hypothesis about permissions.
I see the documentation mentions that maintenance is done occasionally. Does that mean that if its been more than an hour since Kopia was last invoked, it would try to run quick maintenance the next time I invoke Kopia?
How often should I run quick + full maintenance? Would say once a week or month suffice?
Is it possible for me to pass in a different API key manually for a one-time invocation of the cli to run maintenance tasks? Or would I have to create a separate B2 repository which is identical to my usual one, just differing by the key?
When does the snapshot retention policy get evaluated and old snapshots removed? I'd imagine if it finds snapshots to delete, it would need to modify / delete files and would fail without keys with sufficient permissions. If I were to guess, this happens during maintenance?
Answer by Jarek Kowalski 11:54 PM May 9 2021
@dickson Tan some answers:
The quick maintenance needs to run reasonably frequently to ensure good performance - without Kopia will become slower and slower as the number of index files grows. How frequently - it depends on your usage
The full maintenance is only needed to free up disk space when data from old snapshots is no longer needed, if you don’t care about that, you can completely disable it (kopia maintenance set --enable-full false) and/or run manually whenever you feel like it (kopia maintenance run --full)
You can prevent maintenance from running as part of all the other commands by passing undocumented --no-auto-maintenance parameter to every command
To use different set of access keys for maintenance, it’s easiest to create a separate config file (kopia repo connect --config-file /path/to/some-other-file.config b2 --access-key=XX and run maintenance manually by explicitly setting that: kopia --config-file=/path/to/some-other-file.config maintenance run
Snapshot retention is evaluated after each snapshot and this is when old snapshots are marked for deletion (this does not delete actual files but only writes ‘tombstone’ markers for existing snapshots so they don’t show up in the list) . Only full maintenance ultimately gets rid of the contents.
"I'm more concerned about something crawling into the machine and finding the credentials to the repository somewhere in Kopia's config and using that to mass delete files." - @Neurrone
A shared document has been created to facilitate a transparent discussion logging design choicese, tweaking to changes in Kopia and for drafting documentation. Feel free support and comment. If you want editing rights to this document, please send me your gmail address on slack.
This issue is based on a conversation on slack that took place on slack from May 8th 2021 onwards.
Related to #711, #743 and #913
User story (B2 specific, but please read the question S3-generic):
As a Kopia user weary of randsomware attacks on backups,
I would like to use S3-compatible object locking
so that files cannot be deleted by randsomware or other malicious actors (that do not have access to my (MFA-protected) S3-storage account).
How would or how could Kopia best relate to a) default B2 bucket retention policies (that lock a file for a set number of days) and b) to lifecycle rules (that allow for deletion after a set number of days to save money on the storage bill)?
The challenge I imagine is in cleaning up unreferenced data at some point. Perhaps a maintainence flag could be set with the amount of days after which the S3-storage provider will delete the file. Repackaging content that is still being referenced in to be deleted files would be trivial I imagine.
## Without using Object Locking / immutability the following should be possible
Note that it should already be possible to use one pair of application keys for backing up and restoring (without delete capability) and another for maintenance (that has delete abilities).
Question by Dickson Tan 7:04 AM May 9 2021
Thanks for confirming my hypothesis about permissions.
I see the documentation mentions that maintenance is done occasionally. Does that mean that if its been more than an hour since Kopia was last invoked, it would try to run quick maintenance the next time I invoke Kopia?
How often should I run quick + full maintenance? Would say once a week or month suffice?
Is it possible for me to pass in a different API key manually for a one-time invocation of the cli to run maintenance tasks? Or would I have to create a separate B2 repository which is identical to my usual one, just differing by the key?
When does the snapshot retention policy get evaluated and old snapshots removed? I'd imagine if it finds snapshots to delete, it would need to modify / delete files and would fail without keys with sufficient permissions. If I were to guess, this happens during maintenance?
Answer by Jarek Kowalski 11:54 PM May 9 2021
@dickson Tan some answers:
The quick maintenance needs to run reasonably frequently to ensure good performance - without Kopia will become slower and slower as the number of index files grows. How frequently - it depends on your usage
The full maintenance is only needed to free up disk space when data from old snapshots is no longer needed, if you don’t care about that, you can completely disable it (kopia maintenance set --enable-full false) and/or run manually whenever you feel like it (kopia maintenance run --full)
You can prevent maintenance from running as part of all the other commands by passing undocumented --no-auto-maintenance parameter to every command
To use different set of access keys for maintenance, it’s easiest to create a separate config file (kopia repo connect --config-file /path/to/some-other-file.config b2 --access-key=XX and run maintenance manually by explicitly setting that: kopia --config-file=/path/to/some-other-file.config maintenance run
Snapshot retention is evaluated after each snapshot and this is when old snapshots are marked for deletion (this does not delete actual files but only writes ‘tombstone’ markers for existing snapshots so they don’t show up in the list) . Only full maintenance ultimately gets rid of the contents.