generate_presigned_url is 3x as slow as boto2 for s3 resources

[adepue]

When calling generate_presigned_url on a s3 resource, the resulting calls are 3x as slow as the boto2 equivalent...

In [1]: import timeit
In [7]: setup = '''
   ...: import boto3
   ...: from boto.s3.connection import S3Connection
   ...: b3_conn = boto3.client('s3')
   ...: b2_conn = S3Connection(None, None)
   ...: '''
In [13]: print min(timeit.Timer('b2_conn.generate_url(3600, "GET", bucket="foo-bucket", key="foo-key", query_auth=True)', setup=setup).repeat(7, 10000))
1.95865702629
In [14]: print min(timeit.Timer('b3_conn.generate_presigned_url("get_object", Params={"Bucket": "foo-bucket", "Key": "foo-key"}, ExpiresIn=3600, HttpMethod="GET")', setup=setup).repeat(7, 10000))
5.69024801254

This was run on c3.xlarge instances in us-west-2.

[rayluo]

Confirmed the symptom. Will need to profile it to find the bottleneck.

[jamesls]

@adepue Quick question, is your use case that you're generating a large number of these presigned URLs all at once? From the output above, generating a presigned URL in boto3 looks like it takes 5.69024801254 /10000 or about 0.57 milliseconds which comes out to a rate of 1757 presigned urls per second vs. boto2's 5105 presigned urls a second. We'll look at improving the speed in boto3, but it would be helpful to understand how you're currently using the presigner.

[adepue]

I was generating 12 to 16 of these per page render for links to thumbnail images stored in s3. Under load, with 32 web server processes running on a machine, this generated significant load increase.

[cristianocca]

Any news on this? boto3 generate_presigned_url is quite slow to generate bulk urls.

[asherf]

any news on this issue ?

[hfern]

Any updates?

[riperez]

Any updates on this?

[Dizzzmas]

Ran into this problem today. Found it really frustrating, as it was increasing the response time of my API endpoint by 3-4 seconds which was rather significant. And I was generating URLs for only ~30 objects.

Any news on this being addressed in the near future?

[pabdelhay]

I had the same problem. Solved by setting AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY.

When I migrate my servers to ECS it was no longer necessary to use AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY, since everything was on the same security layer. But when I removed those credentials, my application starts performing low
on generating_presigned_url from S3. The solutions was to rollback and set those credentials (AWS_ACCESS_KEY, AWS_SECRET_ACCESS_KEY) again even it is not mandatory on my new environment.

[jsadn]

I profiled the boto and boto3 code for this, and if you're following this thread - I would assume this issue will not be addressed.
There is not a single thing that can be "fixed" and the ~5x increased latency is simply a result of the entire library being heavier.

Here are my observations:

1. boto3 implements some asynchronous paradigms with "emitting" events at various points during the lifecycle of executing code. Much of the additional overhead for generating presigned links is spent here'
2. boto url-quotes just the values (when you have key-value pairs in your url querystring) while boto3 does this for both keys and values. This doubles the overhead of string manipulation prior to calculating the signature.
3. in order to calculate current epoch, boto does a simple int(time.time()) while boto3 uses more robust datetime manipulation using both datetime and email libraries. These add up.
4. boto3 has additional overhead for logging. Tweaking your logging settings can shave some off, but no matter what it uses more resources than boto.

These, and several more things, all add up to the result documented on this thread.

On my end, we're going to roll our own simple version to generate these presigned urls while trying to eliminate as much overhead as possible from the process.

[eyadfarra]

I faced this issue and resolved it by stopping reinitializing the client each time I wanted to pre-sign the URL. means in case you wish to pre-sign bulk resources only initiate the client once then loop over the resources you want to pre-sign them using the client you initialized at the beginning.

[aBurmeseDev]

Checking in here. Generally if you're not using a session, it takes longer to create the client to load the models. Creating the session would be a one-time hit.
I'd also like to point out some of the workarounds mentioned in this issue here and here. Please let us know if you're still encountering this issue.

[adepue]

I abandoned boto3 for this use case some years ago. I'm not in a position to validate it anymore and am happy with anyone else signing off if closing the issue.

[aBurmeseDev]

The boto3 team has recently announced that the Resource interface has entered a feature freeze and won’t be accepting new changes at this time: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html. We’ll be closing existing feature requests, such as this issue, to avoid any confusion on current implementation status. We do appreciate your feedback and will ensure it’s considered in future feature decisions.

We’d like to highlight that all existing code using resources is supported and will continue to work in Boto3. No action is needed from users of the library.