feat(postgres): allow users to configure postgres to read from S3

[bacongobbler]

If a user wishes to set up an external S3 storage endpoint, they can overwrite the following environment variables:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION
• S3_URL (something like https://s3-us-west-1.amazonaws.com)

TESTING:

• build this image, modify deis-database-rc.yaml to point to this image
• modify deis-database-rc.yaml, injecting the following environment variables:

env:
  - name: AWS_ACCESS_KEY_ID
    value: "myaccesskey"
  - name: AWS_SECRET_ACCESS_KEY
    value: "mysecretkey"
  - name: S3_URL
    value: "https://s3-us-west-1.amazonaws.com"
  - name: AWS_DEFAULT_REGION
    value: "us-west-1"

install Deis, and observe that a db_wal bucket was created in your S3 region with database backups.

closes #33

[jchauncey]

can you give me an example of how to test this?

[bacongobbler]

@jchauncey I haven't fleshed this out fully after thinking this through. I'll write out some documentation on how we can test this in a minute. :)

[krancour]

deis/charts#123 is related. It introduced a secret containing user-provided object storage connection / auth details.

[bacongobbler]

All righty, this should be ready to review. Follow the steps in the OP to test.

[krancour]

Can you maybe ask for bucket name instead and derive the URL from the region and the bucket name?

Reason I ask is because what is supposed to become the Deis-wide config for object-storage options currently seems to ask for bucket name instead of URL:

https://github.com/deis/charts/blob/master/deis-dev/tpl/objectstorage.toml#L29-L35

[bacongobbler]

I think I can fix this up in the future to not require S3_URL at which point boto3 will derive the URL from the region and bucket name, but I don't think I can reasonably do that right now. I can open a ticket for that post-beta though!

[krancour]

Should these key/filesnames change to match the keys that the object storage secret already managed by the chart is using?

[bacongobbler]

No. These keys are used by wal-e and boto3 to set up the connection details to S3/minio. These filenames are utilized by envdir to set up the environment variables required for wal-e/boto3.

[bacongobbler]

FYI @kmala volunteered to refactor this PR on top of deis/charts#123 so we don't have users required to configure object storage in two places

[bacongobbler]

closing in favour of #46