Higher latency when falling back to EC2 credentials when generating large numbers of S3 Presigned URLs

[juphamzipit]

Our application uses S3 Presigned URLs to control access to images stored in a bucket for a user. We recently tried to switch from passing in the credentials to the S3 Client to letting the credential provider chain fall through to using the role on the EC2 instance. In doing this, everything was working fine until the server received more load and then the latency on our requests increased. After rolling back and looking at the code we narrowed down the issue to how the S3 SDK was being used and I was wanting some guidance on how it should be implemented for best performance.

The first issue seemed to be instanciating the S3 client in a C# using block which resulted in creating a new S3 client for every request. From what I can find this is not the ideal way to use the client but instead it should be more like a singleton. Can you please confirm the best practice using an S3 client? Should it be reused as much as possible?

The second issue is with how credentials are gathered by the S3 client. When we pass an access key id/secret key into the S3 client it is much more performant than if we let it use the credentials provider chain fall back to using the role for the EC2 instance. It makes sense that if the client is being disposed of for each request that this would cause performance issues because it has to contact the IMDS for each request, but even after changing the client to a static client, the throughput of the calls was slower. For instance, when calling our API to get 20,000 images at one time the request would take about 3 seconds longer if the credentails are not provided. This test was ran with the S3 client being disposed of each request and by creating a static client that is reused. Using a static client did show a decrease in call time in both cases but the delta between the cases was pretty much the same.

• Using block credentials provided - ~6 seconds
• Using block credentials not provided - ~9 seconds
• Static client credentials provided - ~5 seconds
• Static client credentials not provided - ~8 seconds

SDK Version - 3.7.103.21

Any information you can provide on the proper way to use the S3 client would be greatly appreciated.

[juphamzipit]

Looking into this more, I noticed that the Presigned URL has the url parameter x-amz-security-token when using the temporary credentials generated from the EC2 instance role vs a signature url parameter when using access key id/secret passed into the S3 client. Does generating a presigned url using the temporary credentials just have more overhead vs passing in the access key id/secret?

[dscpinheiro]

For your first question, you should be re-using the service client as much as possible (this re:Invent talk goes into more detail on why: https://youtu.be/t3swORUjuBk?t=216).

For your second question, if explicit credentials (such as an access / secret key pair) are provided, the SDK will use them for the entire service client lifecycle; if they're not provided the SDK will attempt to resolve them when the client is created (so when using multiple clients many of them may be trying to reach IMDS at the same time).

There's a limit of 1024 PPS for each Elastic Network Interface attached to an EC2 instance (see Request throttling here: https://repost.aws/knowledge-center/ec2-linux-metadata-retrieval), and although the SDK will handle retrying any throttled requests, this could explain the numbers you're seeing.

And for your third question, the operation to generate a pre-signed URL is local (i.e. does not need to make any calls to S3), so the token does not generate any significant overhead (but it needs to be included when using temporary credentials).

[juphamzipit]

Regarding question 2 - Shouldn't the SDK be caching the credentials and not have to contact the metadata service for each request?

[dscpinheiro]

The SDK does cache credentials so it doesn't have to reach out to IMDS every time (you can see the logic here: https://github.com/aws/aws-sdk-net/blob/main/sdk/src/Core/Amazon.Runtime/Credentials/FallbackCredentialsFactory.cs) but the initial attempt may still result in throttles.

You can also enable debug logging to see exactly what the SDK is doing:

using Amazon;
using Amazon.Runtime;
using Amazon.Runtime.CredentialManagement;
using Amazon.S3;

AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetricsFormat = LogMetricsFormatOption.JSON;
AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogMetrics = true;

var s3Client = new AmazonS3Client();
var response = await s3Client.ListBucketsAsync();
Console.WriteLine(response.Buckets.Count);