Support for STREAMING-UNSIGNED-PAYLOAD-TRAILER requests

[vagaerg]

When we initially implemented the signature validation, we checked 3 main cases:

• W3C_CHUNKED - requests that contain a payload that is chunked using purely HTTP chunking
• AWS_CHUNKED - requests that contain a payload that is chunked using AWS Chunking (determined by a Content-Encoding that contains aws-chunked, as well as potentially other encodings), where each chunk contains a signature (See SigV4 Streaming)
• AWS_CHUNKED_IN_W3C_CHUNKED - an AWS Chunked payload (i.e., with signatures), itself chunked in HTTP chunks (which do not contain signatures)

There is an additional type, STREAMING-UNSIGNED-PAYLOAD-TRAILER - see the docs. This is described as

Use this when sending an unsigned payload over multiple chunks. In this case you also have a trailing header after the chunk is uploaded.

While this already existed by the time we implemented the first version of the proxy, we never saw such requests being sent by standard SDKs, but through some testing now we have seen some sent by aws-cli.

Despite the aws-chunked content encoding, this payload looks exactly like a standard W3C chunked payload with no chunk signatures.

I think we can ignore the trailer header, which should be safe as its value is not signed - since it is sent after the body, it cannot be included in the authorization header (and, thus is not a signed header). We should however check how Jersey handles trailer headers, but given they are part of the HTTP spec I'd hope it handles them correctly.

I believe we can handle these cases gracefully by simply treating cases where we get a request with aws-chunked content encoding with any STREAMING-UNSIGNED-PAYLOAD- hash headers as being W3C Chunked

[vagaerg]

Another example with a larger payload and multiple chunks, using boto3