S3: Upload with storage class

[ThinkChaos]

I saw that in the README glacier is said to be supported via lifecycle rules. I was wondering if there's a reason why the upload's storage class is not configurable.

From a quick look at s3manager's API reference it should be quite easy: UploadInput has a StorageClass field, so AWSS3Backend.Upload could easily forward a configurable value to that.

Is there a reason this hasn't been done? Would you be open to a PR to add this?

[someone1]

I can't recall the exact reasoning at the time, but thinking aloud here - to simplify the scope of this application.

Can you tell me why and/or how different it would be to just set a lifecycle rule (or default class) to the bucket you upload to vs having zfsbackup-go set the class for you?

It also isn't necessarily a common use case that you'd want to keep recent backups on Glacier. My guess is most users would adopt storage classes to align with RTO objectives, and can rely on lifecycle rules accordingly (e.g. Hot -> Warm after 90 days -> Cold after 180 days -> Frozen after 365 days).

I wouldn't be opposed to a PR I just would like to understand your use case better.

[ThinkChaos]

Thank you for the reply. You're right to ask for my use-case, I understand not wanting to bloat your software.

My use-case is storing personal backups offsite. This is data from my home NAS, so I don't mind the extra delay in case of restoring as the contents are only for me and I don't have any hard requirements on availability. Besides I would only access these in case something goes really wrong (fire, water, too many dead disks, or whatever) because I already have enough local ZFS snapshots for when I fuck something up.

As I see it there are two advantages to storing directly to glacier:

• cost savings: the data never hits standard storage so costs go down.
• simplicity: no prerequisites for my S3 zfsbackup-go config to work as expected; and I always know where the backups are.
  Reducing moving parts is a big one for me because having my config being the only thing needed to get this back up and running would be really nice as I know I'd most likely forget the rest.

AFAICT storing directly to glacier isn't possible with a lifecycle rule. I don't have a lot of experience with S3 so I might have missed something.
I hadn't heard of changing the default storage class before, and from what I found it's not possible. Again, you know more about S3 than me so I could be wrong or just have misunderstood what you mean.

[someone1]

I see - I think this should be a pretty straight forward change, I can probably whip it up later this week, but feel free to take a stab at it.

The option should probably be passed in as a env var: AWS_S3_STORAGE_CLASS which we'd pass directly into UploadInput.StorageClass as you've highlighted it should go.

I confused S3 and GCS a bit, the latter lets you set the default storage class at the bucket level, and the former doesn't - but you can set up a lifecycle rule with 0 days to get close to this.

[sjagoe]

I've just come across your project, and plan to start using it soon.

An option to set the S3 storage class on upload would be useful, as the minimum time for a lifecycle rule is 30 days, so for every upload you incur at least one month of "full" cost for S3-Standard before it can be moved to another storage class.

My use case for this would be to immediately upload with the S3-Standard-IA storage class, which is about 1/2 the cost of S3 Standard, with a slightly higher retrieval cost (and charged at a minimum of 30 days storage).

Edit: I'm also just a home user backup up my NAS, which is largely where backups of other systems in the house go, plus a few home services, databases, etc.