-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Random test failures on some tests on master #1256
Comments
Yup, sadly this is a bit of a known problem. It's particularly common with slower test environments because several of the tests run with short timeouts or are expected to run in a fairly straightforward environment. These are tricky to fix. Because they're timing related, the only way to get them to behave the way we actually want in a reliable manner is to write a complete harness for the test environment that mocks out almost any I/O or timing syscalls. This is quite the endeavour, so we normally just tolerate some minor flakiness, especially as it tends only to be exposed in limited environments. |
Ok, thanks for the explanation. I'm surprised that I'd be running into these failures, though. I'm running on a MacBookPro 2.8Ghz i7. I'm also doing some other things on the machine, but it's usually fine for development. I've also been testing in a xenial vagrant box and am seeing similar issues. One last thing I've noticed is that the pypy tests seem to crash during the ssl tests. The last line of output looks like this:
I've tried excluding that test and the next set of ssl tests exhibit the same behavior. |
Interesting. I haven't been able to reproduce this at all. What PyPy are you using? |
pypy2-5.7.1 installed by pyenv. I also had to install cryptography specially to have it able to include opensslv.h from macports:
|
I switched to pypy-5.4.1, installed by pyenv, and the crash doesn't happen, so it seems it's an issue with the newer pypy. |
I've been working on a patch and was trying to get a baseline for the test suites using tox on the master branch. I'm getting random hangs and test failures with the dummyserver tests. They will sometimes pass and sometimes fail. Here's an example of a failure:
I'm also seeing random failures with test/contrib/test_securetransport.py and some of the other ssl based tests.
The text was updated successfully, but these errors were encountered: