InvalidSignatureException on retry after a call is throttled

[anthonybgale]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Signatures don't appear to be valid for the retry of a throttled call (at least in macie:associateMemberAccount).

I had a case where I was consistently seeing an InvalidSignatureException error for the 52nd call in a loop after some minor maintenance and dependency upgrades. After debugging for a while I found that I consistently get the following output for the failed call using a log level of aws.LogDebugWithHTTPBody in my client config:

-----------------------------------------------------
2020/04/16 20:16:05 DEBUG: Request Amazon Macie/AssociateMemberAccount Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: macie.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/0.20.0 (go1.14.1; linux; amd64) exec-env/AWS_Lambda_go1.x
Content-Length: 34
Authorization: AWS4-HMAC-SHA256 Credential=redacted/20200416/us-east-1/macie/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=redacted
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20200416T201605Z
X-Amz-Security-Token: redacted
X-Amz-Target: MacieService.AssociateMemberAccount
Accept-Encoding: gzip

{"memberAccountId":"redacted"}
-----------------------------------------------------
2020/04/16 20:16:05 DEBUG: Response Amazon Macie/AssociateMemberAccount Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Length: 97
Content-Type: application/x-amz-json-1.1
Date: Thu, 16 Apr 2020 20:16:05 GMT
X-Amzn-Requestid: 4cc2265b-fde3-4e5d-8bc7-b0cd13cef6f2


-----------------------------------------------------
2020/04/16 20:16:05 {"__type":"ThrottlingException","message":"Rate exceeded for the key = aws-account:redacted"}
2020/04/16 20:16:05 DEBUG: Request Amazon Macie/AssociateMemberAccount Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: macie.us-east-1.amazonaws.com
User-Agent: aws-sdk-go/0.20.0 (go1.14.1; linux; amd64) exec-env/AWS_Lambda_go1.x
Content-Length: 34
Amz-Sdk-Invocation-Id: 672A9DD0-0588-4A2F-9FDC-0E338ACCEBB7
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=redacted/20200416/us-east-1/macie/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=redacted
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20200416T201605Z
X-Amz-Security-Token: redacted
X-Amz-Target: MacieService.AssociateMemberAccount
Accept-Encoding: gzip

{"memberAccountId":"redacted"}
-----------------------------------------------------
2020/04/16 20:16:05 DEBUG: Response Amazon Macie/AssociateMemberAccount Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 400 Bad Request
Content-Length: 229
Content-Type: application/x-amz-json-1.1
Date: Thu, 16 Apr 2020 20:16:05 GMT
X-Amzn-Requestid: db86761e-6fcd-4575-a9bf-851d35f8525d


-----------------------------------------------------
2020/04/16 20:16:05 {"__type":"InvalidSignatureException","message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
---[ REQUEST POST-SIGN ]-----------------------------

You can see that retry does not appear to have a valid signature.

Version of AWS SDK for Go?
This seems to occur in all recent versions (tested 0.17.0 - 0.20.0)

Version of Go (go version)?
1.14.1

To Reproduce (observed behavior)
I know this happens with macie:associateMemberAccount, not sure about other calls. To reproduce, make a call fast enough to get throttled. Observe the response from AWS when the call is retried.

Expected behavior
I expected the macie client to generate valid signatures for retries, as well as initial requests.

Additional context

This may be a duplicate of this issue.

[anthonybgale]

It appears that the signature algorithm doesn't play well with the addition of the Amz-Sdk-Invocation-Id and Amz-Sdk-Request headers added to the signed headers on retry. If I blacklist those headers here then I don't have the issue with retries after throttle.

[jasdel]

Thanks for creating this issue @anthonybgale I think the issue you are seeing is related to #521 like you mentioned. Something is modifying the request headers after the signature is getting created. Lets use #521 to track this issue.

[jasdel]

We've released this fix as a tagged released, v0.21.0. You should be able to update to pull in these fixes and updates.