s3 setup for archival storage

[ivotron]

When deploying a cluster it would be great to have an option that setup infra support for archival storage feature in redpanda. Briefly this would be:

1. s3 bucket
2. query credentials, urls, etc...
3. deploy redpanda.yml with archival settings from (1) and (2)

@Lazin do you have the set of configuration elements that we'll need? I thought they were in configuration.h but I don't see them in there. Maybe they haven't merged yet?

[Lazin]

Yes, this is not merged yet. The options are:

  archival_storage_enabled: true
  archival_storage_s3_access_key: <...>
  archival_storage_s3_secret_key: <...>
  archival_storage_s3_region: eu-north-1
  archival_storage_s3_bucket: redpandabackup2
  archival_storage_upload_interval: 10000
  archival_storage_max_connections: 5

developer mode should be enabled, you also have to specify AWS security credentials, bucket should be created in advance, versioning and encryptions are not needed, upload_interval is in milliseconds (5 or 10 seconds is OK), max_connections sets number of simultaneous S3 uploads per-shard, values in 1-10 range should be fine

[ivotron]

thanks @Lazin! that info should be good to get me started (I think)

[dotnwat]

@ivotron so we have three modes

1. ductape local
2. ducktape cluster
3. punisher

We're handling (1) with an extra docker container running a local s3 endpoint. For (2) and (3) I assume these share a common method for deployment?

[Lazin]

I renamed archival_storage_upload_interval to archival_storage_reconciliation_interval_ms. It has a sensible default so there is no need to specify it explicitly anywhere. Will create PR with doc tomorrow.

[vuldin]

Is this ticket tracking adding shadow indexing feature, or is this for some other (possibly more specific) issue? I didn't want to create an additional ticket if it wasn't needed.

[vuldin]

Created a new ticket for SI on AWS: #87

Associated PR: #86