You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee=Noneclosed_at=<Date2020-06-25.06:31:52.469>created_at=<Date2020-06-17.12:08:15.951>labels= ['library', '3.10', 'performance']
title='HTTPResponse.read with amt is slow'updated_at=<Date2020-06-25.06:31:52.469>user='https://github.com/bmerry'
I've run into this on 3.8, but the code on Git master doesn't look significantly different so I assume it still applies. I'm happy to work on a PR for this.
When http.client.HTTPResponse.read is called with a specific amount to read, it goes down this code path:
if amt is not None:
# Amount is given, implement using readinto
b = bytearray(amt)
n = self.readinto(b)
That's pretty inefficient, because
bytearray(amt) will first zero-fill some memory
tobytes() will make an extra copy of this memory
if amt is big enough, it'll cause the temporary memory to be allocated from the kernel, which will also zero-fill the pages for security.
A better approach would be to use the read method of the underlying fp.
I have a micro-benchmark (that I'll attach) showing that for a 1GB body and reading the whole body with or without the amount being explicit, performance is reduced from 3GB/s to 1GB/s.
For some unknown reason the requests library likes to read the body in 10KB chunks even if the user has requested the entire body, so this will help here (although the gains probably won't be as big because 10KB is really too small to amortise all the accounting overhead).
Output from my benchmark, run against a 1GB file on localhost:
Why, are you getting significantly different results?
Just in case it's confusing, the results are reported as A ± B MB/s, where A is the mean and B is the standard deviation of the mean. So it's about 3GB/s when no length if passed, or 1GB/s when a length is passed.