New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explicitly close requests connections. #2147

Merged
merged 2 commits into from Jul 16, 2017

Conversation

Projects
None yet
5 participants
@sebastienvercammen
Member

sebastienvercammen commented Jul 16, 2017

Description

requests releases connections to the pool when the response's content is consumed, but we don't need the content - we only parse the response headers.

I've also set the requests to stream=True so we save time by not downloading the response content.

Motivation and Context

Performance and reduce resources.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement

Checklist:

  • My code follows the code style of this project.
@jagauthier

This worked well for my Linux test. I tested with 2k,5k,and 11k proxies. I cannot produce any errors, and this should be an improvement for open sockets, and less requests.

These proxies are also totally sporadic. I tested with 126 good proxies (7 good ones duplicated) and my tests were also good, and successful. I think it takes a moment longer per request to close the two connections, but that's alright!

@SiteEffect

Tested 10, 100, 1000 proxies with various concurrencies 👍 .

@sebastienvercammen sebastienvercammen merged commit 78c28dd into RocketMap:develop Jul 16, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@sebastienvercammen sebastienvercammen deleted the sebastienvercammen:keepalive branch Jul 16, 2017

SiteEffect added a commit to Alderon86/RocketMap that referenced this pull request Jul 17, 2017

Explicitly close requests connections. (RocketMap#2147)
* Explicitly close connections.

* Enable stream so it doesn't download the content.

ixotech1 added a commit to ixotech1/RocketMap that referenced this pull request Jul 24, 2017

Explicitly close requests connections. (RocketMap#2147)
* Explicitly close connections.

* Enable stream so it doesn't download the content.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment