Setting any value to PutObject's poContentEncoding breaks signing

[mgajda]

Attempting to run this snippet:

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Network.AWS
import Network.AWS.S3
import Network.AWS.S3.PutObject
import Control.Lens

bucketName = "test.migamake.com"

main = do
  env  <- newEnv Discover
  runResourceT $ runAWS env $
    within Frankfurt $ do
      content <- chunkedFile 8192 "public/index.html"
      send $ set poContentType     (Just "text/html")
           $ set poContentEncoding (Just "identity")
           $ putObject bucketName (ObjectKey "index.html") content

Results in signing error:

*** Exception: ServiceError (ServiceError' {_serviceAbbrev = Abbrev "S3", _serviceStatus = Status {statusCode = 403, statusMessage = "Forbidden"}, _serviceHeaders = [("x-amz-request-id","C211E7670B81BA51"),("x-amz-id-2","VdRQD1mnawJ2iXGc+CYJnPNsH8ZPtIiWcUqZUieBuOh4SBZXZNEjCSZnpQvXS7/UTElB3c68Qbw="),("Content-Type","application/xml"),("Transfer-Encoding","chunked"),("Date","Sun, 28 Jul 2019 08:50:02 GMT"),("Connection","close"),("Server","AmazonS3")], _serviceCode = ErrorCode "SignatureDoesNotMatch", _serviceMessage = Just (ErrorMessage "The request signature we calculated does not match the signature you provided. Check your key and signing method."), _serviceRequestId = Just (RequestId "C211E7670B81BA51")})

This breaks workaround for #536.

Note that normally there may be multiple content encodings put into a single stance, but it seems that now we have aws-chunked hardwired and it interferes with attempt to assign value here.

I prepared a small example of this issue:
https://github.com/mgajda/aws-empty-content-encoding-issue/tree/assign-content-encoding

[mgajda]

It seems that:

1. This parameter is incorrectly modeled as a String, instead of comma-separated ordered list of strings.
   https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding
   Normally aws-chunked would be prepended to such list (since they are applied in order on the client.)
2. Process of signing adds this header, and thus breaks itself if there is duplicate header?

[mgajda]

Got confirmation from AWS, that the empty content encoding is known issue.
Recommended workaround is to set poContentEncoding, which is currently a problem in amazonka.
Or setting response-content-encoding that will override Content-encoding of the response from S3.

[finlay]

I think we have run into this problem too. Strangely, something seems to have change last weekend! It was working fine up to September 13, 2019, and it wasn't working on September 16.

Did something change ? Did the AWS S3 signing rules change ?

Below is snippet where we call putObject:

       True -> do                                                  
            f <- chunkedFile 32768 srcPath                          
            _ <- send $ putObject (BucketName $ storeBucket config) (fromString destPath) f
            return ()          

[finlay]

Okay, I managed to figure out a few things.

1. There was no change of our code, or of the AWS S3 signing rules. However, I think our network traffic is being modified, with Content-Type and Content-Encoding headers being modified by network providers. I believe this was the immediate problem.

2. The Amazonka library will include the Content-* headers in the v4 signature. This is not necessary according to AWS docs. My understanding is that it makes it more secure to sign more headers. However, if network providers mess with your headers, it makes things more brittle.

3. The Aws haskell library is more parsimonious about which headers to include in the signature. In particular, it doesn't include the Content-* headers.

4. Switching our code to using the Aws library resolved the problem for us. However, it would be good to fix the Amazonka library to make it handle these cases more carefully.

F

[jchia]

@finlay I started getting the same SignatureDoesNotMatch error early this week when sending large files in chunks of 512kB using code similar to yours. I haven't change my code and everything had been working fine for many months till late last week. Sending files smaller than the chunk size still works. I'm not sure what changed to break multipart sending.

Regarding what you said about your network provider, aren't S3 requests done over HTTPS instead of HTTP? (I packet-captured my S3 session and can confirm that it was using HTTPS.) Since HTTPS is encrypted, I'm not sure how your network providers can mess with your headers unless your HTTPS session is compromised.

Since the error shows up even without using poContentType and poContentEncoding and only appeared in the last few days, I'm not sure @finlay and I have the same issue as the one originally described in the title. I think the pertinent issue for us is #546.

[sarashino]

@finlay and @jchia, as my comrade had asked this issue to AWS support desk in Japan, we’ve got an information that they modified S3 to reject a request which Transfer-Encoding header field “and” Content-Length header field compliant with RFC7230: “A sender must not send a Content-Length header field in any message that contains a Transfer-Encoding header field”, since Sep 16, 2019. Thus I believe these matters are originated by restriction of HTTP standardization.

I hope this helps.

[endgame]

Possibly related (since they affect S3 request signing)? #547 #596