AmazonSecurityTokenServiceClient AssumeRoleAsync duplicated user agent header causes SignatureDoesNotMatch error

[Xriuk]

Describe the bug

I'm signing Amazon SP-API requests with my AWS role, I have a service which has two (or more) instances running in parallel via Tasks each instance tries to assume the same AWS role and one of them fails.
I inspected the requests and the one failing has the user-agent header which gets duplicated.
I had no problems until I ran 2 services in parallel.

Here's my code:

using var STSClient = new AmazonSecurityTokenServiceClient(AccessKey, SecretKey);

var response = await STSClient.AssumeRoleAsync(new AssumeRoleRequest() {
  RoleArn = RoleArn,
  DurationSeconds = 3600,
  RoleSessionName = Guid.NewGuid().ToString()
}, cancellationToken);

Here's the request that succeeds:

POST https://sts.amazonaws.com/ HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.101.8 aws-sdk-dotnet-core/3.7.105.2 .NET_Core/6.0.13 OS/Microsoft_Windows_10.0.19043 ClientAsync
amz-sdk-invocation-id: 48d8fc02-08e2-472d-b1fe-fc512c52db73
amz-sdk-request: attempt=1; max=5
Host: sts.amazonaws.com
X-Amz-Date: 20230308T160729Z
X-Amz-Content-SHA256: 37bd71e2071571eebdbb3d377a9d44d17cd851b3263eacf1968bfc78ed69f582
Authorization: AWS4-HMAC-SHA256 Credential=<credential>/20230308/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=71b12f9e23a4d834f9c5beed230448b8467a59e9c5c849f88a0a3d239998f448
Content-Length: 178
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=AssumeRole&DurationSeconds=3600&RoleArn=<role>&RoleSessionName=<value>&Version=2011-06-15

And here's the one failing:

POST https://sts.amazonaws.com/ HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.101.8 aws-sdk-dotnet-core/3.7.105.2 .NET_Core/6.0.13 OS/Microsoft_Windows_10.0.19043 ClientAsync aws-sdk-dotnet-coreclr/3.7.101.8 aws-sdk-dotnet-core/3.7.105.2 .NET_Core/6.0.13 OS/Microsoft_Windows_10.0.19043 ClientAsync
amz-sdk-invocation-id: 24ae31e9-fc48-462a-8ff0-3e2c02595758
amz-sdk-request: attempt=1; max=5
Host: sts.amazonaws.com
X-Amz-Date: 20230308T160729Z
X-Amz-Content-SHA256: 4b00a7a3dbb5728f4b39748bd4759fa755d973ae4fcc7b6e1639eaa3a318a461
Authorization: AWS4-HMAC-SHA256 Credential=<credential>/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=e65dfd28b77e61c241d1ab11bc7d63fd972aef2ba50e3f6831de7e92362962c0
Content-Length: 178
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=AssumeRole&DurationSeconds=3600&RoleArn=<role>&RoleSessionName=<value>&Version=2011-06-15

As you can see the user-agent header is repeated twice

Expected Behavior

The request should have the correct user-agent header

Current Behavior

The request fails with the response:

HTTP/1.1 403 Forbidden
x-amzn-RequestId: fb2343ea-6dc2-40b5-b510-173c7999ebe3
Content-Type: text/xml
Content-Length: 431
Date: Wed, 08 Mar 2023 16:07:55 GMT

<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
  </Error>
  <RequestId>fb2343ea-6dc2-40b5-b510-173c7999ebe3</RequestId>
</ErrorResponse>

Reproduction Steps

Try to run two (or more) Tasks in parallel which try to assume the same role

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.SecurityToken 3.7.101.8

Targeted .NET Platform

.NET 6

Operating System and version

Windows 10

[ashishdhingra]

Reproducible intermittently (only if network monitoring tool is running in background) using the below code (using AWSSDK.SecurityToken 3.7.101.21):

using Amazon.SecurityToken;
using Amazon.SecurityToken.Model;

string accessKey = "<<access-key>>";
string secretKey = "<<secret-key>>";
string roleArn = "<<arn-of-role-to-assume>>";

var parallelTasks = Enumerable.Range(0, 6).Select(i => 
{
    return Task.Run(async () =>
    {
        using AmazonSecurityTokenServiceClient STSClient = new AmazonSecurityTokenServiceClient(accessKey, secretKey);

        AssumeRoleResponse response = await STSClient.AssumeRoleAsync(new AssumeRoleRequest()
        {
            RoleArn = roleArn,
            DurationSeconds = 3600,
            RoleSessionName = Guid.NewGuid().ToString()
        });
        Console.WriteLine($"Iteration {i}, response: {response.Credentials}");
    });
});

await Task.WhenAll(parallelTasks);

Launched Fiddler in background to capture network traffic.

Failed Fiddler trace:

• Request:

POST https://sts.amazonaws.com/ HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.101.21 aws-sdk-dotnet-coreclr/3.7.101.21 aws-sdk-dotnet-core/3.7.105.15 .NET_Core/6.0.14 OS/Microsoft_Windows_10.0.19044 ClientAsync
amz-sdk-invocation-id: 032a28d8-c7b7-40a7-a441-efaa74bef2ec
amz-sdk-request: attempt=1; max=5
Host: sts.amazonaws.com
X-Amz-Date: 20230309T195149Z
X-Amz-Content-SHA256: 8877bdffc4c54a3b8dc6a7c0ea0a826fec043222ceb1ade72e1bd68cd6eb40e0
Authorization: AWS4-HMAC-SHA256 Credential=AKIASA6NRDFTVVGUMCXE/20230309/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9331a83484279438245ab98e70f0ca3c0b1e1e0c61affbfe51fd50cb324d96e6
Content-Length: 184
Content-Type: application/x-www-form-urlencoded; charset=utf-8

Action=AssumeRole&DurationSeconds=3600&RoleArn=arn%3Aaws%3Aiam%3A%3A139480602983%3Arole%2Ftestassumerole-ashdhin&RoleSessionName=49483c05-77a2-4ace-a5e1-082d1e6868cf&Version=2011-06-15

• Response

HTTP/1.1 403 Forbidden
x-amzn-RequestId: efb68360-3a8b-4011-a534-e8c6636df267
Content-Type: text/xml
Content-Length: 431
Date: Thu, 09 Mar 2023 19:51:49 GMT

<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
  </Error>
  <RequestId>efb68360-3a8b-4011-a534-e8c6636df267</RequestId>
</ErrorResponse>

However, the issue is reproducible only if network monitoring tool (Fiddler in this case) is running in background to capture network traffic. Needs review with the team.

@Xriuk Please share the following:

• What network monitoring tool you are using?
• Do you encounter the issue if you disable network monitoring in background?

Thanks,
Ashish

[Xriuk]

Yes, I confirm that I had Fiddler open in the background, also without it open it seems that everything works.

[ashishdhingra]

s-sdk-dotnet-coreclr/3.7.101.21 aws-sdk-dotnet-coreclr/3.7.101.21

@Xriuk Thanks for your response. Fiddler is a network proxy tool, so I'm unsure if there is an issue with fiddler which is messing up the user agent header. Please confirm if this issue is closed since it's not feasible to troubleshoot Fiddler issue.

Thanks,
Ashish

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.