dagpool_test busy loop on CPython 3.7 epool/select hubs

[temoto]

While executing dagpool_test, random tests will burn CPU in kernel. Observed as

./venv-37/bin/nosetests -svx tests/dagpool_test.py
tests.dagpool_test.test_spawn_collision_spawn ... ok
tests.dagpool_test.test_spawn_multiple ... ok
tests.dagpool_test.test_spawn_many ... ok
tests.dagpool_test.test_wait_each_all ... ok
tests.dagpool_test.test_kill ... ok
tests.dagpool_test.test_post_collision_preload ... ok
tests.dagpool_test.test_post_collision_post ... ok
tests.dagpool_test.test_post_collision_spawn ... ok
tests.dagpool_test.test_post_replace ... ok
tests.dagpool_test.test_getitem ... ok
tests.dagpool_test.test_waitall_exc ... ok
tests.dagpool_test.test_propagate_exc ... ^C^C^\[1]    26649 quit       ./venv-37/bin/nosetests -svx tests/dagpool_test.py
   23.65s user 0.06s system 99% cpu 23.727 total

Note system 99%.

• Nose process is not responding to SIGINT (KeyboardInterrupt).
• Particular tests to cause busy loop: test_propagate_exc, test_wait_each_exc.
• They pass alone, problem occurs while running whole dagpool_test.
• Environment: Linux, python 3.7, epoll and select hubs

Example of build failed for this reason:
https://travis-ci.org/eventlet/eventlet/jobs/351626233

@nat-goodspeed could you look what is the problem here?

Similar busy loop problem with test_accept_deflate_ext_context_takeover_13 (tests.websocket_new_test.TestWebSocketWithCompression). Difference is: under strace websocket tests segfault with

recvfrom(9, "", 2, 0, NULL, NULL)       = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x99} ---

and dagpool doesn't

write(2, "tests.dagpool_test.test_propagate_exc ... ", 42tests.dagpool_test.test_propagate_exc ... ) = 42
clock_gettime(CLOCK_MONOTONIC, {3917231, 687934179}) = 0
clock_gettime(CLOCK_MONOTONIC, {3917231, 687991532}) = 0
clock_gettime(CLOCK_MONOTONIC, {3917231, 688159907}) = 0
clock_gettime(CLOCK_MONOTONIC, {3917231, 688235242}) = 0
clock_gettime(CLOCK_MONOTONIC, {3917231, 688261079}) = 0
^C--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL} ---

[temoto]

Current status: 2018-03-10 on Linux CPython 3.7 Eventlet may cause busy loop, use in production is not recommended, subscribe to this issue to get updates.

[nat-goodspeed]

Sorry, I'm at a conference all this week. I will look into it next week.

[nat-goodspeed]

Are you building Python 3.7 from source? I tried to find it on python.org, but it seems to be unavailable as yet. That makes it tough to debug: I don't know how to obtain the interpreter version with which this fails.

If Python 3.7 has not yet been released, isn't it possible that what you're seeing is a bug in Python 3.7?

[temoto]

@nat-goodspeed it is there, https://www.python.org/downloads/release/python-370b3/

3.7-dev supplied by Travis also produces this problem.

[nirik]

We are hitting this in Fedora as well: https://bugzilla.redhat.com/show_bug.cgi?id=1594248

If I comment out test_wait_each_exc() then it hangs at test_post_get_exc()

If I disable both of those it crashes at: (tests.websocket_new_test.TestWebSocketWithCompression) ... /var/tmp/rpm-tmp.mxJJ4m: line 34: 11070 Segmentation fault (core dumped) nosetests-3.7 -v
(which may be a different issue)

[nat-goodspeed]

Ah, so Python 3.7 was officially released yesterday. There goes my hope that it was a bug in the pre-release version. I haven't forgotten about this, really!

[vstinner]

I created an upstream bug report: https://bugs.python.org/issue33996

I'm able to reproduce the crash without eventlet, only using greenlet.

[nat-goodspeed]

Thank you!
Doesn't that mean the fix for Python 3.7 would be in greenlet, though, rather than the Python interpreter? Or am I missing the point?

[vstinner]

Doesn't that mean the fix for Python 3.7 would be in greenlet, though, rather than the Python interpreter?

I wrote a script to reproduce the bug only using greenlet. Maybe it's a bug in greenlet, maybe in Python. Honestly, I don't know at this point. At least, it doesn't seem to be a bug in eventlet.

[vstinner]

The bug is in greenlet, not in Python: python-greenlet/greenlet#131

[hroncok]

With @vstinner'spatch, this issue is solved, yet eventlet still fails with 3.7:

#502

[vstinner]

With @vstinner'spatch, this issue is solved, yet eventlet still fails with 3.7: (...)

I suggest to open a new issue to track these issues, since they are different (unrelated) ;-)

[nat-goodspeed]

Or at least change the name of this one... dagpool_test doesn't seem to be the culprit after all.

[nat-goodspeed]

None of the test failures in #502 seem related to dagpool_test. Closing.