rustic restore/rustic prune and cold storage

[kapitainsky]

As clarified earlier rustic does not support today partial repo check due to issues with partial cold storage files warm up.

Does the same apply to restore and prune?

Let's say I have many TB in cold storage (S3 Glacier) and I want to restore only small subset of data. Will rustic request warming up all repo? Or only required files? With cold storage like Glacier it is important as there is cost attached to warm up operations.

And the same for prune. Will only files needed be warmed up? Or all repo?

These are points I would love to clarify before I start fighting with aws-cli warm-up command and all Glacier setup as outlined here - #692 (comment)

[aawsome]

prune and restore are fully compatible with cold storage, i.e. only the needed data is requested for warm-up.
For prune, however, the default is not to repack data pack files (see --repack-cacheable-only) - for cold storages the trade-off between keeping more data than needed and requesting and repacking partly used pack files normally goes into the direction of keeping more unused data.
Completely unused pack files are deleted by prune, of course.

[kapitainsky]

For prune, however, the default is not to repack data pack files (see --repack-cacheable-only) - for cold storages the trade-off between keeping more data than needed and requesting and repacking partly used pack files normally goes into the direction of keeping more unused data.

but can this default be overwritten? And to prune pack files? Of course it will take time (waiting for warm-up) etc.

[aawsome]

Sure, by using --repack-cacheable-only=false. Then packs will be requested for warm-up

[kapitainsky]

One more clarification needed.

So if I run:

rustic prune --repack-cacheable-only=false --dry-run

does it mean that warm-up will be requested?

I imagine I would like to check first if it is worth to run prune without any extra cost.... but as I understand it is not possible? At least not directly... as I can always use fake warm-up command to do it.

[aawsome]

due to issues with partial cold storage files warm up.

Actually it is just not implemented. But implementation is a bit more complex than the "hot data only" case because cold storage needs special treatment.

[kapitainsky]

Thank you for all clarifications!

rustic rocks!

[aawsome]

One thing about the warm-up: There are currently two possibilities:

• specify a time for warm-up-wait. Then rustic will start the warm-up, wait for the given time and then processes with the command (restore/prune).
• [UPDATE: This is wrong. see below] run the command in dry-run mode: This will anyway start the warm-up. After you successfully verified that the warm-up was successfuly, you can re-ran the command (with warm-up-wait=0s).

Note that rustic assumes that it can access warmed-up files exactly with the same repository/storage "coordinates" which it also uses for writing data to the storage. I.e. after the warm-up files need to be accessible under s3:/bucket/repo-path as given in repository.

[kapitainsky]

Yeap. This is all clear and fine. This is how Glacier works anyway so no issue here. Just moving object from one storage class to another without changing location.

[kapitainsky]

• run the command in dry-run mode: This will anyway start the warm-up.

Hmmmmm.... I do not think it is nice... --dry-run should not really do anything like this IMO. Does it really need to read files from cold storage to produce results? Especially when in cases like Glacier it has cost attached.

[aawsome]

Ah, sorry, I was wrong. In fact --dry-run does not warm-up. This was something in older rustic versions but was reverted. I wanted a workflow where users can check if everything is warmed-up and then run the command. But this can be easily done by first specifying a large warm-up-wait time and simply cancel rustic while waiting. Then, when everything is warmed up, just re-ran with out warm-up-wait.

[kapitainsky]

Perfect. I think I will just use warm-up-wait set to whatever is max quoted by cold storage provider - if it is 12h let it be even if files are ready faster.

Most important is that I can use --dry-run to asses and test what I am planning to do without paying for warm-ups

[aawsome]

Ah, and there is so far no feedback from the storage implemented, i.e. rustic cannot wait until the storage reports that everything needed is warmed up.

[kapitainsky]

Yeah this I guess is not easy when you rely on rclone for all connectivity. But it is good enough to be 100% practical - even though requires maybe a bit scripting.

[aawsome]

This is especially not easy given that there is no real standard for it. Some cold storage may require polling for the status, others give an HTTP response where the wait time is encapsulated in some string which needs to be parsed,...
So, yes, future rustic versions may be able to support such things, but currently the pragmatic approach IMO also works.

[kapitainsky]

Yes I see that it could be hard. And I imagine that sometimes some files can be ready in 10 minutes but others in 10 hours. It is definitely not show stopper for using cold storage - at the end this is storage where restore operations are not supposed to happen as business as usual. Not even sure I would spend time on trying to make it better to be honest given many other more important things.

What is missing here is just clear documentation with some real life examples. Limitations and caveats are not a problem - problem is when it is not clear what they are.

There are bits and pieces all over github but none is really complete. Hopefully I will play with it and can try to help with documenting my journey:)

[kapitainsky]

BTW - from other posts and configs examples I see that you are using OVH cold archive. What is your experience? Have you tried to restore?

Their pricing is very competitive - especially restore. This is part I am not 100% from their website. They say €0.005 which would be €5 for restoring 1TiB which is extremely competitive to Amazon Glacier (it would cost €100+). Are there some hidden costs with OVH and cold restore?

I am looking for home for my long term archive (but sometimes I have to pull some data out) and OVH looks a bit too good to be true:)

[kapitainsky]

I use IDrive S3 for hot storage. I tried all - B2, Wasabi, AWS etc. And IDrive is the most affordable feature reach solution I found. It is just what I call fair pricing they offer. Others charge premium for stuff I probably do not need anyway.

$4 per TB/month for pay as you go and $2.5 if paid for yearly package. 3x paid stored data egress, no transactions, no minimum retention or any other such nonsense. Multiple hosting location in US/Europe and Asia. Support S3 lock, versioning etc.

Since I am using them only what could make me move is somebody offering cheaper price with at least similar service.

For cold storage I am ok with AWS Deep Glacier.

[nickchomey]

I'll have to check out idrive! The transactions is what gives me concern for backblaze - especially since I believe rustic works by breaking up files into smaller chunks. Still, the fast-list rclone option should minimize costs. I'll try to test it out soon and report back.

[kapitainsky]

I've been using it now for more than a year - multiple geographies. It is nothing fancy but working S3 storage for fair money.
I was a bit concerned about who IDrive guys are as never heard about them before - it turned out that this is actually company in storage business for 12 years longer than Backblaze. They used to do computer backup stuff - similar to crashplan etc. I guess at some stage they decided to extend their offering by S3 storage and could leverage their existing infrastructure etc.

[aawsome]

I took a look at OVH Cold Archive, out of interest. See https://help.ovhcloud.com/csm/de-public-cloud-storage-cold-archive-getting-started?id=kb_article_view&sysparm_article=KB0034659, here are my conclusions:

• It uses standard S3 to upload and download and archiving/restoring can be scripted using "quasi"-standard s3 Tiering configuration.
• Archiving can be done only for the whole bucket. Also when restoring, the whole bucket will be restored
• Minimum data is 1TB - less data will be priced as 1 TB.

So, my conclusion is that this type of cold storage is not usable as "daily backup target". You can, of course, put a restic/rustic repository there, archive the whole repository and when needed "restore" the whole repository and access it. This however also means that you cannot use restic or rustic on an archived repository - even listing snapshots (rustic snapshots) or contained files (rustic ls) wouldn't work.
You could use rustic`s hot/cold repository and only archive the cold repository. In that case all read-only commands which wouldn't need file contents were fully working - but again, to restore file contents, you would first need to restore the complete cold repository.

My personal conclusion is that I don't have a use-case for this. I personally prefer cold services which can freeze or warm-up specific pack files; this allows "daily" backups and individual restore with only few data warmed-up.

[kapitainsky]

Thank you for investigating it and sharing your findings. Very useful.