Skip to content

worker continue die with gevent loop engine #68

Closed
liujuncn opened this Issue Nov 29, 2012 · 8 comments

5 participants

@liujuncn

uWSGI 1.4.2
gevent 1.0rc1

INI file:
[uwsgi]
gid = pgy
uid = pgy
socket = 127.0.0.1:3131
master = true
processes = 1
loop = gevent
async = 50
enable-threads = true
harakiri = 60
socket-timeout = 30
disable-logging = true
daemonize = /home/pgy/log/uwsgi/%n.log
virtualenv = /home/pgy/venv
module = runserver
callable = app

uwsgi log:
*** Starting uWSGI 1.4.2 (64bit) on [Thu Nov 29 00:11:56 2012] ***
compiled with version: 4.6.3 on 28 November 2012 10:45:02
os: Linux-3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012
nodename: pgy-web1
machine: x86_64
clock source: unix
detected number of CPU cores: 8
current working directory: /home/pgy/pgysocial
detected binary path: /usr/local/bin/uwsgi
your processes number limit is 63631
your memory page size is 4096 bytes
*** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
detected max file descriptor number: 10240

  • async cores set to 50 - fd table size: 10240 lock engine: pthread robust mutexes uwsgi socket 0 bound to TCP address 127.0.0.1:3131 fd 3 Python version: 2.7.3 (default, Aug 1 2012, 05:25:23) [GCC 4.6.3] Set PythonHome to /home/pgy/venv Python main interpreter initialized at 0x241e910 python threads support enabled your server socket listen backlog is limited to 100 connections mapped 1117728 bytes (1091 KB) for 50 cores *** Operational MODE: async *** WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x241e910 pid: 6932 (default app) *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 6932) spawned uWSGI worker 1 (pid: 6939, cores: 50) *** running gevent loop engine [addr:0x45d140] *** !!! uWSGI process 6939 got Segmentation Fault !!! *** backtrace of 6939 *** uwsgi(uwsgi_backtrace+0x25) [0x442bb5] uwsgi(uwsgi_segfault+0x21) [0x442c91] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fe4251e94a0] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7fe4256b2e6a] /usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fe4256b37d7] uwsgi(python_call+0x24) [0x44e6f4] uwsgi(uwsgi_gevent_current_wsgi_req+0x1d) [0x45e89d] uwsgi(warn_pipe+0xa) [0x43fdea] /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fe4251e94a0] /lib/x86_64-linux-gnu/libpthread.so.0(send+0x22) [0x7fe4267cc282] /usr/lib/libpython2.7.so.1.0(+0x81ea7) [0x7fe4255f3ea7] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4f18) [0x7fe4256033b8] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(+0xd5047) [0x7fe425647047] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7fe42560246d] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7fe4256045eb] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(PyObject_CallFunctionObjArgs+0xb0) [0x7fe4256f8a40] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x17fe) [0x7fe4255ffc9e] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7fe4256038c0] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(+0x13136c) [0x7fe4256a336c] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fe4256b37d7] uwsgi(python_call+0x24) [0x44e6f4] uwsgi(uwsgi_request_wsgi+0xe1) [0x450a81] uwsgi(py_uwsgi_gevent_request+0xc2) [0x45eb12] /usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4f18) [0x7fe4256033b8] /usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7fe4255ce605] /usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7fe4255ce7bd] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] /usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7fe4256971cf] /usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7fe4256b2e83] *** end of backtrace *** DAMN ! worker 1 (pid: 6939) died :( trying respawn ... Respawned uWSGI worker 1 (new pid: 7008)
@unbit
Owner
unbit commented Nov 29, 2012

does it happens on the first request or as soon as the server starts ?

@liujuncn

As I observed, it happens on the first request starts.

After I switch to preforking mode, the web app can run normally. (BTW, the OS is 3.2.0-29-generic #46-Ubuntu 12.04 LTS server)

Same code will run normally with gevent loop engine under Centos 2.6.32-220.13.1.el6.x86_64.

@dmalenko

I also experience similar crashes on first request after nobody touched the server for some time:

uwsgi(uwsgi_backtrace+0x25) [0x4452b5]
uwsgi(uwsgi_segfault+0x21) [0x445391]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f0f1ca4d4a0]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x3a) [0x7f0f1d14fe6a]
/usr/lib/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7f0f1d1507d7]
uwsgi(python_call+0x24) [0x451f94]
uwsgi(uwsgi_gevent_current_wsgi_req+0x1d) [0x46228d]
uwsgi(warn_pipe+0xa) [0x44228a]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f0f1ca4d4a0]
/lib/x86_64-linux-gnu/libpthread.so.0(write+0x2d) [0x7f0f1e4a5ccd]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_real_write+0x9a) [0x7f0f1656b87a]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_flush+0x1b) [0x7f0f1656bd4b]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(net_write_command+0x1a2) [0x7f0f1656bf12]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(cli_advanced_command+0x10c) [0x7f0f165691ac]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.18(mysql_close+0x59) [0x7f0f165674f9]
/opt/DAILY/local/lib/python2.7/site-packages/_mysql.so(+0x4ff4) [0x7f0f16a79ff4]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x53a5) [0x7f0f1d0a0845]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x131720) [0x7f0f1d140720]
/usr/lib/libpython2.7.so.1.0(+0x141f08) [0x7f0f1d150f08]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7f0f1d09f46d]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c8c0) [0x7f0f1d06b8c0]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x2b2a) [0x7f0f1d09dfca]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5420) [0x7f0f1d0a08c0]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x131134) [0x7f0f1d140134]
/usr/lib/libpython2.7.so.1.0(PyObject_GetIter+0x3e) [0x7f0f1d144cae]
/usr/lib/libpython2.7.so.1.0(+0xdb7f0) [0x7f0f1d0ea7f0]
/usr/lib/libpython2.7.so.1.0(+0x81322) [0x7f0f1d090322]
/usr/lib/libpython2.7.so.1.0(+0x141f08) [0x7f0f1d150f08]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3fcd) [0x7f0f1d09f46d]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]
/usr/lib/libpython2.7.so.1.0(+0x5c7bd) [0x7f0f1d06b7bd]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x1251cf) [0x7f0f1d1341cf]
/usr/lib/libpython2.7.so.1.0(PyObject_Call+0x53) [0x7f0f1d14fe83]
/usr/lib/libpython2.7.so.1.0(+0x10ee94) [0x7f0f1d11de94]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b33) [0x7f0f1d09efd3]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x614b) [0x7f0f1d0a15eb]
/usr/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x855) [0x7f0f1d06b605]

The system is also on Ubuntu 12.04 LTS using uWSGI 1.4.9 and gevent 1.0rc2.

@unbit
Owner
unbit commented Mar 18, 2013

how you are making mysql gevent friendly ?

@dmalenko

You are right, I did not do anything specific to make mysql play nicely with gevent. So, I guess, my comment can be disregarded.

@prymitive
Collaborator

Is this issue still valid? Does it work with 1.9?

@gilles
gilles commented May 8, 2013

Seems I'm running in a similar issue with 1.9, not sure it's the same though

*** HARAKIRI ON WORKER 4 (pid: 142962, try: 1) ***
*** backtrace of 142962 ***
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_backtrace+0x25) [0x453795]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(what_i_am_doing+0x30) [0x4538d0]
/lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7fb4a3cd40b0]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x33) [0x7fb4a3d974d3]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x22475) [0x7fb499be6475]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(ev_run+0x284) [0x7fb499bedde4]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/src/gevent/gevent/core.so(+0x2a909) [0x7fb499bee909]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5e75) [0x7fb4a43098d5]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3913) [0x7fb4a1dba913]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3340) [0x7fb4a1dba340]
/srv/www/app.kwarter.com/releases/20130506213646/virtualenv/local/lib/python2.7/site-packages/greenlet.so(+0x3d96) [0x7fb4a1dbad96]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x47) [0x7fb4a43d0b37]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa0499) [0x7fb4a433e499]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1bfe) [0x7fb4a430565e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x4a00) [0x7fb4a4308460]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x81a) [0x7fb4a439d2fa]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xff4a6) [0x7fb4a439d4a6]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(+0xa106d) [0x7fb4a433f06d]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_Call+0x4e) [0x7fb4a43cfd8e]
/usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0(PyObject_CallMethod+0xea) [0x7fb4a42eaa3a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x46fe4a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_ignition+0xba) [0x453c0a]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi(uwsgi_start+0x1488) [0x4558e8]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c306]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fb4a3cbeea5]
/srv/www/app.kwarter.com/current/virtualenv/bin/uwsgi() [0x41c331]
*** end of backtrace ***
[uwsgi]
socket = /tmp/app.socket
chown-socket = www-data
master = true
processes = 8
harakiri = 30
gevent = 5
gevent-monkey-patch = true
#threaded-logger = true => does not make any difference
stats = 127.0.0.1:9191
max-requests = 10000
wsgi-file = wsgi.py
lazy = true
#enable-threads = true => does not make any difference
@unbit
Owner
unbit commented May 8, 2013

no, it is a simple harakiri triggered because your app is blocked. You can get more infos about why your app was blocked adding the tracebacker (by default you only get the C traceback)

@unbit unbit closed this Sep 19, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.