Cassandra backups and missing features

[cscetbon]

Describe the problem/challenge you have
A few features are missing in order to backup objects in a Cassandra database without recreating a new cluster. Same in order to have the ability to use different retentions for specific objects. That's what this issue is about.

Describe the solution you'd like

• In order to have different retention per object we would need a way to specify includes/excludes rules for restic backups. The question is where to put this information. Adding this information as annotations on pods would mean they would be global, except if the annotation has a specific format by being a json string for instance, like

{
 "k1": {"include": "data/k1/*/snapshots/"},
 "k2/t1": {"include": "data/k2/t1/snapshots/"} 
...
}

We would then need a way to say which restic rule we want to apply for a specific backup.

• Hooks should have access to those params in order for instance to know what to snapshot for a Cassandra backup. We could also want to keep some snapshots local and use them to restore from instead of restoring from s3 using restic. That could speed up restore after a an operation that corrupted data and decrease the MTTR.

• Restoring is done by creating a new cluster, however with Cassandra there is the ability to restore files in place and ask Cassandra to refresh its internal state to see the new content of the files. We would need the ability to restore files in place but also to have hooks on restore to trigger the refresh.

I'm probably missing other requirements but we can flesh them out.

[eleanor-millman]

Closing because this will hopefully be covered by Astrolabe API (perhaps 2022).