storage: allow configuration of storage region for cloud storage (PRO…

…JQUAY-3082)

Boto3 behaves unexpectedly when the resource client is not set to use
the correct region. Boto3 can't seem to correctly set the
X-Amz-Credential header when generating presigned urls if the region
name is not explicitly set, and will always fall back to us-east-1.
To reproduce this:
- Create a bucket in a different region from us-east-1 (e.g
eu-north-1)
- Create a boto3 client/resource without specifying the region
- Generate a presigned url

This seems to be a DNS issue with AWS that only happens shortly after
a bucket has been created, and resolves itself eventually. To
workaround this, one can explicitly specify the bucket endpoint.
Ref:
- boto/boto3#2989
- https://stackoverflow.com/questions/56517156/s3-presigned-url-works-90-minutes-after-bucket-creation

storage/cloud.py

@@ -731,6 +731,7 @@ def __init__(
         s3_bucket,
         s3_access_key=None,
         s3_secret_key=None,
+        region_name=None,
         # Boto2 backward compatible options (host excluding scheme or port)
         host=None,
         port=None,
@@ -739,7 +740,10 @@ def __init__(
     ):
         upload_params = {"ServerSideEncryption": "AES256"}
         connect_kwargs = {"config": Config(signature_version="s3v4")}
-        if host or endpoint_url:
+        if region_name is not None:
+            connect_kwargs["region_name"] = region_name
+            connect_kwargs["endpoint_url"] = "https://s3.{region}.amazonaws.com".format(region=region_name)
+        elif host or endpoint_url:
             connect_kwargs["endpoint_url"] = endpoint_url or _build_endpoint_url(
                 host, port=port, is_secure=True
             )