New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gevent 1.3.0 crashing uwsgi #1212

Closed
kunalg opened this Issue May 16, 2018 · 6 comments

Comments

Projects
None yet
2 participants
@kunalg

kunalg commented May 16, 2018

  • gevent version: 1.3.0
  • Python version: cpython 2.7.6, uwsgi version: 2.0.15
  • Operating System: Ubuntu 14.04.5 LTS Server (Linux #196-Ubuntu SMP Wed May 2 15:51:34 UTC 2018 GNU/Linux)

Description:

My app uses gevent for coroutines and I use uwsgi as the app server.
It started crashing and going into restart loop since upgrading gevent to v1.3.0.

Same code with same environment works fine with gevent v1.2.2.

Here is the stacktrace of the crash:

WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x148f580 pid: 32076 (default app)
WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0x148f580 pid: 32075 (default app)
Python auto-reloader enabled
*** running gevent loop engine [addr:0x491820] ***
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 527, in gevent._greenlet.Greenlet.spawn
  File "src/gevent/greenlet.py", line 247, in gevent._greenlet.Greenlet.__init__
  File "src/gevent/greenlet.py", line 133, in gevent._greenlet._extract_stack
ValueError: call stack is not deep enough
!!! uWSGI process 32075 got Segmentation Fault !!!
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 527, in gevent._greenlet.Greenlet.spawn
  File "src/gevent/greenlet.py", line 247, in gevent._greenlet.Greenlet.__init__
  File "src/gevent/greenlet.py", line 133, in gevent._greenlet._extract_stack
ValueError: *** backtrace of 32075 ***
uwsgi(uwsgi_backtrace+0x2e) [0x46b9be]
uwsgi(uwsgi_segfault+0x21) [0x46bd81]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f84f4821cb0]
uwsgi() [0x491e42]
uwsgi(uwsgi_ignition+0x11d) [0x46bf5d]
uwsgi(uwsgi_worker_run+0x2dd) [0x47084d]
uwsgi() [0x470e3f]
uwsgi(_start+0) [0x41f17e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f84f480cf45]
uwsgi() [0x41f1a7]
*** end of backtrace ***
call stack is not deep enough
!!! uWSGI process 32076 got Segmentation Fault !!!
*** backtrace of 32076 ***
uwsgi(uwsgi_backtrace+0x2e) [0x46b9be]
uwsgi(uwsgi_segfault+0x21) [0x46bd81]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0) [0x7f84f4821cb0]
uwsgi() [0x491e42]
uwsgi(uwsgi_ignition+0x11d) [0x46bf5d]
uwsgi(uwsgi_worker_run+0x2dd) [0x47084d]
uwsgi() [0x470e3f]
uwsgi(_start+0) [0x41f17e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f84f480cf45]
uwsgi() [0x41f1a7]
*** end of backtrace ***
DAMN ! worker 1 (pid: 32075) died, killed by signal 11 :( trying respawn ...
Respawned uWSGI worker 1 (new pid: 32089)
DAMN ! worker 2 (pid: 32076) died, killed by signal 11 :( trying respawn ...
Respawned uWSGI worker 2 (new pid: 32090)

What I've run:

It's a django application using django v1.4.3.

Relevant part from uwsgi.ini:

[uwsgi]
vacuum			= true

max-requests		= 5000
http			= :8000
gevent			= 1000
gevent-early-monkey-patch = 1

workers			= 2
listen			= 100

lazy-apps		= true
single-interpreter	= true
enable-threads		= true

## dev mode - reload app
py-autoreload		= 2
die-on-term         	= 1
@jamadden

This comment has been minimized.

Member

jamadden commented May 16, 2018

ValueError: call stack is not deep enough

Appears to be some limitation in the Python interpreter, at least as uwsgi embeds it. gevent is just calling sys._getframe() with no arguments, so we're not asking for any particular stack depth.

As a workaround, can you disable track_greenlet_tree?

@kunalg

This comment has been minimized.

kunalg commented May 16, 2018

Setting track_greenlet_tree via env variable in uwsgi.ini as:
env = GEVENT_TRACK_GREENLET_TREE=0
seems to help. 👍

I'll open an issue with uwsgi project then.

Thanks,
Kunal

@jamadden

This comment has been minimized.

Member

jamadden commented May 16, 2018

There is an issue: unbit/uwsgi#1790

I see what's going on, uwsgi is issuing a C call to functions that are implemented in C in gevent 1.3.0. There are no intervening Python stack frames, hence the error. gevent can handle that gracefully.

@kunalg

This comment has been minimized.

kunalg commented May 16, 2018

Okay, great.

So, what are the next steps? any way I can help?

Thanks a lot for the help 👍

jamadden added a commit that referenced this issue May 16, 2018

Catch the ValueError that sys._getframe() can raise when there are no…
… Python frames.

This can happen in the embedded case, when the CPython C api is used to directly call Greenlet.spawn, which itself is now implemented in C.

Fixes #1212.
@jamadden

This comment has been minimized.

Member

jamadden commented May 16, 2018

I have a fix in #1213. If you wanted to try running a source checkout and verifying that would be nice, but the fix is so simple that's probably not worth the effort.

@kunalg

This comment has been minimized.

kunalg commented May 16, 2018

Yup, tested it just now - for the kicks of it 😀
Works as expected..

thanks a lot for taking a look at this.

Thanks,
Kunal

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment