Ability to specify the upload_storage_class in S3 bucket configuration

[arpad-m]

Currently we move data to the intended storage class via lifecycle rules, but those are a daily batch job so data first spends up to a day in standard storage.

Therefore, make it possible to specify the storage class used for uploads to S3 so that the data doesn't have to be migrated automatically.

The advantage of this is that it gives cleaner billing reports.

Part of https://github.com/neondatabase/cloud/issues/11348

[github-actions]

2766 tests run: 2645 passed, 0 failed, 121 skipped (full report)

---

Code coverage* (full report)

• functions: 28.1% (6480 of 23046 functions)
• lines: 47.0% (46055 of 98007 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
4608a4a at 2024-04-24T15:57:36.177Z :recycle:}

[koivunej]

I don't think there is any situation where we wouldn't want to do this, so approving. It can later be made a bucket setting, but cannot see a reason why.

[problame]

Nit: the PR description doesn't explain that this is mostly to have cleaner billing reports: https://github.com/neondatabase/cloud/issues/11348#issuecomment-2069597349

[arpad-m]

Hmm the error is legit (link):

 INFO request{method=POST path=/v1/tenant/c67b90954f2d8c2f44ead4ea780ac5b9/timeline/ request_id=98c65ac2-2682-43fb-9c2c-3cbae5efd42c}:timeline_create{tenant_id=c67b90954f2d8c2f44ead4ea780ac5b9 shard_id=0000 timeline_id=19a2fc0428d3ac78084e3e0558aeaca1 lsn=None pg_version=Some(14)}: persist_initdb_tar_zst failed, will retry (attempt 0): upload initdb dir for 'c67b90954f2d8c2f44ead4ea780ac5b9 / 19a2fc0428d3ac78084e3e0558aeaca1': Failed to upload data of length 1853469 to storage path RemotePath("tenants/c67b90954f2d8c2f44ead4ea780ac5b9/timelines/19a2fc0428d3ac78084e3e0558aeaca1/initdb.tar.zst"): service error: unhandled error (InvalidStorageClass): Error { code: "InvalidStorageClass", message: "Invalid storage class.", aws_request_id: "17C909D09A9C3B5A" }

[arpad-m]

On slack we agreed to make it configurable instead (and leave it off by default) to help people who use S3 clones like minio, e.g. selfhosters, but also various local development workflows in Neon (like the failing CI job). We'll then set the parameters in production pageserver configuration.