S3 broken due to deprecated URLs

[AriFordsham]

AWS S3 has two styles of S3 URLs:

• path-style: https://s3{-<region>}.amazonaws.com/<bucket-name>/<object-or-command>
• virtual-hosted style: https://<bucket-name>.s3{.<region>}.amazonaws.com/<object-or-command>

Path-style is officially deprecated but still supported ([1], [2], [3]). It seems the way they are supporting this in eu-west-2 is by returning a HTTP 301 redirect response.

This means Network.AWS.S3 is completely broken, at least for my bucket in this region.

Example:

GHC session: (formatted for clarity)

*Network.AWS Network.AWS.S3> runResourceT . runAWS env $ send (getObject "<bucket>" "<object>")

*** Exception:
    ServiceError (
        ServiceError' {
            _serviceAbbrev = Abbrev "S3",
            _serviceStatus = Status {
                statusCode = 301,
                statusMessage = "Moved Permanently"
            },
            _serviceHeaders = [
                ("x-amz-bucket-region","eu-west-2"),
                ("x-amz-request-id","*****"),
                ("x-amz-id-2","******"),
                ("Content-Type","application/xml"),
                ("Transfer-Encoding","chunked"),
                ("Date","Wed, 26 May 2021 12:24:49 GMT"),
                ("Server","AmazonS3"),
                ("Connection","keep-alive")
            ],
            _serviceCode = ErrorCode "PermanentRedirect",
            _serviceMessage = Just (ErrorMessage "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."),
            _serviceRequestId = Just (RequestId "******")}
    )

If I intercept the request, I see it is:

https://s3.amazonaws.com/<bucket-name>/<object-name>

Requesting this URL using Postman yields a '301 Moved Permanantly' response with the following body:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>PermanentRedirect</Code>
    <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
    <Endpoint><bucket-name>.s3.amazonaws.com</Endpoint>
    <Bucket><bucket-name></Bucket>
    <RequestId>*****</RequestId>
    <HostId>***********</HostId>
</Error>

Correcting the URL to

https://<bucket-name>.s3.eu-west-2.amazonaws.com/<object-name>

Yields a correct 200 response.

Configuring Service will allow me to change the endpont, but this doesn't work, as it still appends the bucket name to the URL, as so:

https://<bucket-name>.s3.eu-west-2.amazonaws.com/<bucket-name>/<object-name>

Which is incorrect and yields a 404 NoSuchKey.

This issue could be solved in one of two ways:

• Changing amazonka-s3 to use the supported virtual-hosting style URLs
• Following 301 redirects

[AriFordsham]

Current service description from AWS (https://github.com/boto/botocore/blob/develop/botocore/data/s3/2006-03-01/service-2.json) is wrong.

I have filed an issue boto/botocore#2397 to see what is going on.

[AriFordsham]

Boto swallows the incorrect definitions and handles them in code: boto/botocore#52 (Initial PR from 2013, implementation has evolved since)

[AriFordsham]

This is also happening with us-west-2. I haven't yet had amazonka-s3 working correctly. Am I doing something wrong, or is this package completely broken right now?

[avanov]

S3 and GetObject works for me from this checkout - 14bc324 It also seems to be not yet released on Hackage - #562 (comment)