Skip to content

Commit

Permalink
Add new environ variable for s3 storage class
Browse files Browse the repository at this point in the history 
WALE_S3_STORAGE_CLASS, defaulting to STANDARD.  valid values: STANDARD | REDUCED_REDUNDANCY | GLACIER | STANDARD_IA | ONEZONE_IA | INTELLIGENT_TIERING | DEEP_ARCHIVE
  • Loading branch information
@ftzdomino @fdr
ftzdomino authored and fdr committed Feb 9, 2021
1 parent 28498ff commit bec6f21
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 1 deletion.
2 changes: 2 additions & 0 deletions README.rst
View file
Expand Up @@ -71,6 +71,8 @@ AWS S3 and Work-alikes
Optional:

* WALE_S3_ENDPOINT: See `Manually specifying the S3 Endpoint`_
* WALE_S3_STORAGE_CLASS: One of: STANDARD (default), REDUCED_REDUNDANCY,
GLACIER, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, DEEP_ARCHIVE
* AWS_SECURITY_TOKEN: When using AWS STS
* Pass ``--aws-instance-profile`` to gather credentials from the
Instance Profile. See `Using AWS IAM Instance Profiles`.
Expand Down
3 changes: 2 additions & 1 deletion wal_e/blobstore/s3/s3_util.py
View file
Expand Up @@ -54,7 +54,8 @@ def uri_put_file(creds, uri, fp, content_type=None, conn=None):
if content_type is not None:
k.content_type = content_type

k.set_contents_from_file(fp, encrypt_key=True)
storage_class = os.getenv('WALE_S3_STORAGE_CLASS', 'STANDARD')
k.set_contents_from_file(fp, encrypt_key=True, headers=storage_class)

This comment has been minimized.

Sign in to view

Copy link
@dfaubion-zinc

dfaubion-zinc Mar 15, 2021 •
edited

This is broken. set_contents_from_file takes headers as an object. storage_class is a string. Because encrypt_key is specified, this always results in an error, but the error gets supressed. I suspect #416 and #415 may be related to this.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/wal_e/operator/backup.py", line 198, in database_backup
    ret_tuple = self._upload_pg_cluster_dir(
  File "/usr/local/lib/python3.8/dist-packages/wal_e/operator/backup.py", line 490, in _upload_pg_cluster_dir
    uri_put_file(self.creds,
  File "/usr/local/lib/python3.8/dist-packages/wal_e/worker/worker_util.py", line 12, in uri_put_file
    return blobstore.uri_put_file(creds, uri, fp,
  File "/usr/local/lib/python3.8/dist-packages/wal_e/blobstore/s3/s3_util.py", line 58, in uri_put_file
    k.set_contents_from_file(fp, encrypt_key=True,  headers=storage_class)
  File "/usr/local/lib/python3.8/dist-packages/boto/s3/key.py", line 1219, in set_contents_from_file
    headers[provider.server_side_encryption_header] = 'AES256'
TypeError: 'str' object does not support item assignment

This comment has been minimized.

Sign in to view

Copy link
@fdr

fdr Mar 15, 2021 •
edited

Member

I wonder if something changed in the driver at some point. I have never used this feature myself. (or maybe a refactor went bad)

This comment has been minimized.

Sign in to view

Copy link
@dfaubion-zinc

dfaubion-zinc Mar 16, 2021

Probably bad refactor. I looked into downgrading my boto to fix the issue, but it takes a dict as far back as 2014.

This comment has been minimized.

Sign in to view

Copy link
@fdr

fdr via email Mar 18, 2021

Member

This comment has been minimized.

Sign in to view

Copy link
@dfaubion-zinc

dfaubion-zinc Mar 18, 2021 •
edited

I think the answer is to replace headers=storage_class with
headers={"x-amz-storage-class": storage_class}, changing it to this is enough to allow backups to finish again.

This comment has been minimized.

Sign in to view

Copy link
@fdr

fdr Mar 18, 2021

Member

Oh I see, this is new functionality I merged. Okay, will patch.

This comment has been minimized.

Sign in to view

Copy link
@fdr

fdr Mar 18, 2021

Member

Alright, it's in, how does it feel
return k


Expand Down

0 comments on commit bec6f21

Please sign in to comment.