Unify timeouts in tests#1717
Conversation
sethmlarson
left a comment
sethmlarson
left a comment
There was a problem hiding this comment.
Thanks! Preliminary comments for you.
| # want to use a long timeout, even more so on CI where tests can be really slow | ||
| # 3. To test our timeout logic by using two different values, eg. by using different | ||
| # values at the pool level and at the request level. | ||
| SHORT_TIMEOUT = 0.001 |
There was a problem hiding this comment.
Looks like Travis (which is faster than AppVeyor usually) timed out on this 0.001 value. Maybe we need to have two separate values for the SHORT_TIMEOUT as well?
There was a problem hiding this comment.
Oh, test_ssl_read_timeout failed, I had not noticed that, sorry. I think the better fix is to switch to LONG_TIMEOUT here. Here's my reasoning.
In this test, we:
- read the headers -- this should not timeout, so using
LONG_TIMEOUTmakes sense - read the body -- this should timeout, so using
SHORT_TIMEOUTmakes sense
However, both timeouts are read timeouts and are controlled by a single value, so we have to compromise. Switching to LONG_TIMEOUT is only a compromise for this test, while bumping SHORT_TIMEOUT would be a compromise for all tests.
What do you think?
There was a problem hiding this comment.
👍 LONG_TIMEOUT works for that test.
| # DeadlineExceededError -> TimeoutError | ||
| with pytest.raises(urllib3.exceptions.TimeoutError): | ||
| self.pool.request("GET", "/sleep?seconds=0.005", timeout=0.001) | ||
| self.pool.request("GET", "/sleep?seconds=0.005", timeout=SHORT_TIMEOUT) |
There was a problem hiding this comment.
If we implement the above comment, can we have the seconds=X call be dynamic? Maybe 2 * SHORT_TIMEOUT?
There was a problem hiding this comment.
Ah yes, great catch! I think I'll continue to use 5 * SHORT_TIMEOUT
The goal is to be able to to tweak the timeout values globally, and to use larger timeouts on CI where running time is less predictable.
A part of the test is not supposed to time out, so a short timeout does not make sense here.
74b9c62 to
8a17b4a
Compare
Codecov Report
@@ Coverage Diff @@
## master #1717 +/- ##
=======================================
Coverage 99.65% 99.65%
=======================================
Files 22 22
Lines 2006 2006
=======================================
Hits 1999 1999
Misses 7 7Continue to review full report at Codecov.
|
|
Rebased on top of master and added two commits for your first two remarks. |
|
I broke the lint, sorry. Will fix as soon as I can. |
A part of the test is not supposed to time out, so a short timeout does not make sense here.
|
I can't say that I've eradicated all flaky tests here, but I find this approach really promising. The separation between the two types of timeouts allows me to reason about the timeouts and justify switching to longer timeouts when it makes sense. This is ready for another review! |
|
Closing/reopening to see if I just got lucky or not. |
0.1 was not enough in this job: https://travis-ci.org/urllib3/urllib3/jobs/604797713
|
The AppVeyor failure is due to my test from #1715, I'll fix it later. The Travis failure is due to 100ms not being enough, I bumped LONG_TIMEOUT to 500ms. |
|
Closing/reopening because it was a good idea the last time. :) |
|
One last try. |
Codecov Report
@@ Coverage Diff @@
## master #1717 +/- ##
=======================================
Coverage 99.65% 99.65%
=======================================
Files 22 22
Lines 2006 2006
=======================================
Hits 1999 1999
Misses 7 7Continue to review full report at Codecov.
|
The goal is to be able to to tweak the timeout values globally, and to use larger timeouts on CI where running time is less predictable.
3 participants
Please review #1712 first, as I included it in this pull request to avoid conflicts in the future.
Closes #1706