Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd a Timeout class #231
Conversation
added some commits
Aug 18, 2013
shazow
reviewed
Aug 18, 2013
added some commits
Aug 18, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 18, 2013
Contributor
I found that timeouts on SSL connections came in a few forms. In the latest pip download the error that matches on a timeout is:
raise ProxyError('Cannot connect to proxy. '
'Socket error: %s.' % e)
In this branch if the read operation times out, a RequestTimeoutError gets raised. However if the read operation starts and does not complete it's a little more ambiguous - was that caused by the timeout on the socket or did either end just decide to close the connection? I'm not sure of the best way to handle it.
|
I found that timeouts on SSL connections came in a few forms. In the latest
In this branch if the read operation times out, a RequestTimeoutError gets raised. However if the read operation starts and does not complete it's a little more ambiguous - was that caused by the timeout on the socket or did either end just decide to close the connection? I'm not sure of the best way to handle it. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 18, 2013
Contributor
I'm happy to merge this in with the existing PR, it's up to you which interface we'd like to have for the code.
|
I'm happy to merge this in with the existing PR, it's up to you which interface we'd like to have for the code. |
shazow
reviewed
Aug 20, 2013
| log = logging.getLogger('urllib3.connectionpool') | ||
| log.setLevel(logging.NOTSET) | ||
| log.addHandler(logging.StreamHandler(sys.stdout)) | ||
| # We need a host that will not immediately close the connection with a TCP | ||
| # Reset. SO suggests this hostname | ||
| TARPIT_HOST = '10.255.255.1' |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pasha-r
Aug 20, 2013
Connection timeout must be tested somehow and the best way is to use some non-existent IP.
I ran into this problem myself and solved it like this:
pasha-r@4a3b45e#L2R79
Which is essentially the same solution.
pasha-r
Aug 20, 2013
Connection timeout must be tested somehow and the best way is to use some non-existent IP.
I ran into this problem myself and solved it like this:
pasha-r@4a3b45e#L2R79
Which is essentially the same solution.
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 20, 2013
Contributor
When you connect to a port on loopback that's not open, the operating system immediately sends a TCP RST packet. Connecting to a 192.168 or a 10.* host does not yield the same reset.
I will look into a mock TCP server that never ACK's. Alternately we could mock out the call to connect() and have the call just sleep, and verify that the connect timeout does what it is supposed to do.
Sample tcpdump traffic on lo0 from a host that's not open (notice the [R.] packet response)
09:36:15.921174 IP6 (hlim 64, next-header TCP (6) payload length: 44) fe80::1.55621 > fe80::1.avt-profile-2: Flags [S], cksum 0xfd35 (incorrect -> 0x1e2f), seq 533239964, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 643765797 ecr 0,sackOK,eol], length 0
09:36:15.921191 IP6 (hlim 64, next-header TCP (6) payload length: 20) fe80::1.avt-profile-2 > fe80::1.55621: Flags [R.], cksum 0xfd1d (incorrect -> 0x0d95), seq 0, ack 533239965, win 0, length 0
Sample tcpdump from a 192.168.* host:
09:41:33.888484 IP (tos 0x10, ttl 64, id 20873, offset 0, flags [DF], proto TCP (6), length 64)
kburke.55711 > 192.168.5.6.avt-profile-2: Flags [S], cksum 0xa14e (correct), seq 2132708649, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 644083291 ecr 0,sackOK,eol], length 0
kevinburke
Aug 20, 2013
Contributor
When you connect to a port on loopback that's not open, the operating system immediately sends a TCP RST packet. Connecting to a 192.168 or a 10.* host does not yield the same reset.
I will look into a mock TCP server that never ACK's. Alternately we could mock out the call to connect() and have the call just sleep, and verify that the connect timeout does what it is supposed to do.
Sample tcpdump traffic on lo0 from a host that's not open (notice the [R.] packet response)
09:36:15.921174 IP6 (hlim 64, next-header TCP (6) payload length: 44) fe80::1.55621 > fe80::1.avt-profile-2: Flags [S], cksum 0xfd35 (incorrect -> 0x1e2f), seq 533239964, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 643765797 ecr 0,sackOK,eol], length 0
09:36:15.921191 IP6 (hlim 64, next-header TCP (6) payload length: 20) fe80::1.avt-profile-2 > fe80::1.55621: Flags [R.], cksum 0xfd1d (incorrect -> 0x0d95), seq 0, ack 533239965, win 0, length 0
Sample tcpdump from a 192.168.* host:
09:41:33.888484 IP (tos 0x10, ttl 64, id 20873, offset 0, flags [DF], proto TCP (6), length 64)
kburke.55711 > 192.168.5.6.avt-profile-2: Flags [S], cksum 0xa14e (correct), seq 2132708649, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 644083291 ecr 0,sackOK,eol], length 0
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Aug 20, 2013
Collaborator
TIL that `10.255.255.1' is a special non-routable address. I presume this will work while you're offline as well, so that's fine with me. :)
shazow
Aug 20, 2013
Collaborator
TIL that `10.255.255.1' is a special non-routable address. I presume this will work while you're offline as well, so that's fine with me. :)
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Aug 20, 2013
Collaborator
I do like where you're going with this. I think this is the "next step" of what @pasha-r was working on. Might be worth leapfrogging straight to the next step if you think we can cover all the same functionality.
Ideally I'd love to have both of you collaborate on this, but I'm not sure how much time @pasha-r has (can you weigh in?).
Would be feasible for you to send your changes against the #145 pull request? I think you have a lot of overlapping stuff that could be reused (such as tests).
|
I do like where you're going with this. I think this is the "next step" of what @pasha-r was working on. Might be worth leapfrogging straight to the next step if you think we can cover all the same functionality. Ideally I'd love to have both of you collaborate on this, but I'm not sure how much time @pasha-r has (can you weigh in?). Would be feasible for you to send your changes against the #145 pull request? I think you have a lot of overlapping stuff that could be reused (such as tests). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pasha-r
Aug 20, 2013
I like the idea of Timeout class.
I'd suggest this: move HTTP(S)Connection to a separate file, like in my last commit:
pasha-r@518d0c3
P.S.: it also solves stylistic problem of renaming Connection classes over and over :)
pasha-r
commented
Aug 20, 2013
|
I like the idea of Timeout class. P.S.: it also solves stylistic problem of renaming Connection classes over and over :) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pasha-r
Aug 20, 2013
I've commited fixes and changes to the #145 PR, so it does actually work now :)
But once I saw your approach with Timeout class, I like it way better.
And you @kevinburke write more comments - always a plus :)
So you may consider some pieces of my PR if you'd like but I'd suggest this PR as generally preferable.
pasha-r
commented
Aug 20, 2013
|
I've commited fixes and changes to the #145 PR, so it does actually work now :) But once I saw your approach with Timeout class, I like it way better. So you may consider some pieces of my PR if you'd like but I'd suggest this PR as generally preferable. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Aug 20, 2013
Collaborator
Fair enough, we can treat this PR as the primary. Would be nice to merge some of the pieces/tests of the other PR where it makes sense. :)
|
Fair enough, we can treat this PR as the primary. Would be nice to merge some of the pieces/tests of the other PR where it makes sense. :) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 20, 2013
Contributor
Will do!
Kevin Burke | 415-723-4116 | www.twilio.com
On Tue, Aug 20, 2013 at 10:54 AM, Andrey Petrov notifications@github.comwrote:
Fair enough, we can treat this PR as the primary. Would be nice to merge
some of the pieces/tests of the other PR where it makes sense. :)—
Reply to this email directly or view it on GitHubhttps://github.com/shazow/urllib3/pull/231#issuecomment-22964123
.
|
Will do! Kevin Burke | 415-723-4116 | www.twilio.com On Tue, Aug 20, 2013 at 10:54 AM, Andrey Petrov notifications@github.comwrote:
|
added some commits
Aug 21, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 21, 2013
Contributor
Ok... I think I addressed the issues you've raised in this PR. @pasha-r, I added you as a contributor to my repo so you should be able to push commits to this branch.
For some reason the unit tests and travis tests are failing with the message "No handlers could be found for logger "nose.plugins.cover"". Tomorrow I will investigate/fix this, merge the tests from @pasha-r's branch, and add more tests to ensure the Timeout.total is doing what we expect it should do.
|
Ok... I think I addressed the issues you've raised in this PR. @pasha-r, I added you as a contributor to my repo so you should be able to push commits to this branch. For some reason the unit tests and travis tests are failing with the message "No handlers could be found for logger "nose.plugins.cover"". Tomorrow I will investigate/fix this, merge the tests from @pasha-r's branch, and add more tests to ensure the Timeout.total is doing what we expect it should do. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
@shazow, I added you to my repo as well |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Aug 21, 2013
Collaborator
The logging thing is just a warning. The failure is that there's <100% coverage. :)
|
The logging thing is just a warning. The failure is that there's <100% coverage. :) |
added some commits
Aug 21, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 22, 2013
Contributor
Okay... I made the following changes
- source_address is Python 2.6 compatible, thanks @pasha-r
- There's now 100% test coverage
- Now that get_request_timeout doesn't take arguments, moved it to be a @Property value, like the Url class.
- Timeout.total defaults to None - it doesn't really make sense to have it be the global timeout as this is covered by the request and connect timeouts. Documentation updated to reflect this
- Legacy timeouts (eg a float or an int) are turned into
Timeout(connect=old_value, request=old_value)instead of justTimeout(request=old_value). This reflects what the old code is doing. There is one additional socket call in httplib.py besides the connect() and the recv(), socket.sendall, but this is nonblocking which means it returns very quickly. - Moved the connection classes into their own file per @pasha-r's suggestion
- Fixed test coverage for 2.6 and 3x
|
Okay... I made the following changes
|
shazow
reviewed
Aug 22, 2013
| # This timeout error does not have a URL attached and needs to inherit from the | ||
| # base HTTPError |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Aug 22, 2013
Collaborator
Maybe it would be better if TimeoutError didn't inherit from RequestError, and then RequestTimeoutError can inherit from both?
shazow
Aug 22, 2013
Collaborator
Maybe it would be better if TimeoutError didn't inherit from RequestError, and then RequestTimeoutError can inherit from both?
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
reviewed
Aug 22, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Aug 31, 2013
Contributor
I think this is ready for another look. @wolever, I destroyed the stop/start interface - I'm not sure it makes much sense as the connect timeout start time is the only piece of information needed to compute the total timeout.
|
I think this is ready for another look. @wolever, I destroyed the stop/start interface - I'm not sure it makes much sense as the connect timeout start time is the only piece of information needed to compute the total timeout. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Anything I can do to move this forward? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Sep 5, 2013
Collaborator
@kevinburke Could you do a rebase or pull the latest master.
@wolever Will you have a chance to take a last look at it? If not, I can do it.
|
@kevinburke Could you do a rebase or pull the latest master. @wolever Will you have a chance to take a last look at it? If not, I can do it. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Lukasa
Sep 5, 2013
Contributor
@shazow Can I ask you to ping me when this gets merged? We'll want to plumb this through from Requests I think. =)
|
@shazow Can I ask you to ping me when this gets merged? We'll want to plumb this through from Requests I think. =) |
added some commits
Sep 5, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Sep 5, 2013
Contributor
@shazow The rebase got a little messy, I just merged master into the branch if it's ok.
I also fixed up the documentation so it actually looks readable and added the exceptions module to the docs.
|
@shazow The rebase got a little messy, I just merged master into the branch if it's ok. I also fixed up the documentation so it actually looks readable and added the exceptions module to the docs. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Sep 9, 2013
Collaborator
@wolever says (but forgot to leave a comment):
The only thing I would specifically change.
The note about DNS:
Since IMO "high load" is fundamentally different from "Python's dns resolver doesn't honour the socket timeout"
|
@wolever says (but forgot to leave a comment): |
shazow
reviewed
Sep 9, 2013
| # Reset the timeout for the recv() on the socket | ||
| read_timeout = timeout_obj.read_timeout | ||
| log.debug("Setting read timeout to %s" % read_timeout) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Sep 10, 2013
Contributor
Yeah I was unsure here as this would be the only way to determine after the fact what the connect() and read() were actually set to..
kevinburke
Sep 10, 2013
Contributor
Yeah I was unsure here as this would be the only way to determine after the fact what the connect() and read() were actually set to..
shazow
reviewed
Sep 9, 2013
| httplib_response = conn.getresponse() | ||
| except SocketTimeout: | ||
| err = ReadTimeoutError(self, url, | ||
| "Read timed out. (read timeout=%s)" % |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Sep 9, 2013
Collaborator
Can we add a it more info into this exception? Like the host at least? (Something similar to Line 369)
shazow
Sep 9, 2013
Collaborator
Can we add a it more info into this exception? Like the host at least? (Something similar to Line 369)
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
reviewed
Sep 9, 2013
shazow
reviewed
Sep 9, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Sep 9, 2013
Collaborator
Bunch of nitpicks. Try to fix as many of them as you can, then let me know when it's ready and I'll merge. :)
Thanks again!
|
Bunch of nitpicks. Try to fix as many of them as you can, then let me know when it's ready and I'll merge. :) Thanks again! |
added some commits
Sep 10, 2013
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Anorov
Sep 10, 2013
Will this new timeout API be reflected in requests at all, @kevinburke? Or is it going to remain strictly at the urllib3 level?
Anorov
commented
Sep 10, 2013
|
Will this new timeout API be reflected in |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Lukasa
Sep 10, 2013
Contributor
@Anorov I would like to reflect it in requests, though I'm not sure in what form that will be just yet.
|
@Anorov I would like to reflect it in |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Sep 11, 2013
Collaborator
I think this is good to go. Want to add yourself to the CONTRIBUTORS.txt before I merge? :)
|
I think this is good to go. Want to add yourself to the |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
pasha-r
Sep 11, 2013
Vanity request!
@kevinburke can you add me as well in contributors to this PR?
Only if you don't mind, of course :)
pasha-r
commented
Sep 11, 2013
|
Vanity request! |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Vanity metrics are the best kind of metrics! |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Fixed in e679ff7 |
added a commit
that referenced
this pull request
Sep 11, 2013
shazow
merged commit f545dcf
into
urllib3:master
Sep 11, 2013
1 check failed
shazow
referenced this pull request
Sep 12, 2013
Closed
Max retries exceeded exception on stale pooled connections #245
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
shazow
Oct 4, 2013
Collaborator
@kevinburke Could you clarify what this test is testing? I'm confused why ssl is being None'd to test the new Timeout behaviour.
|
@kevinburke Could you clarify what this test is testing? I'm confused why ssl is being None'd to test the new Timeout behaviour. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
kevinburke
Oct 4, 2013
Contributor
|
Not sure... I'm AFK until Sunday, will look then.
On Friday, October 4, 2013, Andrey Petrov wrote:
@kevinburke https://github.com/kevinburke Could you clarify what this
test is testing? I'm confused why ssl is being None'd to test the new
Timeout behaviour.
—
Reply to this email directly or view it on GitHubhttps://github.com/shazow/urllib3/commit/b4e5a8f491eb45ef9bfac5b24b118ad022babf89#commitcomment-4252764
.
##
…---
Kevin Burke | 415-723-4116 | www.twilio.com
|
kevinburke commentedAug 18, 2013
You can now specify Timeout values with
in addition to the usual float/int ways of specifying it.
Also adds a test case for the old and the new way of specifying a timeout.