socket is not registered: gthread worker crash #1258

Closed
evanj opened this Issue May 11, 2016 · 0 comments

Projects

None yet

2 participants

@evanj
evanj commented May 11, 2016

I'm using gunicorn 19.4.1, and I haven't been able to figure out how to reproduce this reliably, but I'm seeing the following crash periodically in my gthread workers. This causes the worker to exit and the master to spawn a new one. I also found the following StackOverflow question that mentions running into the same problem: http://stackoverflow.com/questions/26634327/gunicorn-worker-periodically-crashes-socket-is-not-registered

The workaround in that question suggests that the issue could be related to keepalive and connections that were timed out. I could believe that might be related: I seem to hit the issue when my workers are extremely busy.

I will report back if I figure out more details.

Stack trace:

Traceback (most recent call last):
  File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 515, in spawn_worker
    worker.init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 109, in init_process
    super(ThreadWorker, self).init_process()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.run()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 217, in run
    self.murder_keepalived()
  File "/env/local/lib/python2.7/site-packages/gunicorn/workers/gthread.py", line 178, in murder_keepalived
    self.poller.unregister(conn.sock)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 409, in unregister
    key = super(EpollSelector, self).unregister(fileobj)
  File "/env/local/lib/python2.7/site-packages/gunicorn/selectors.py", line 244, in unregister
    raise KeyError("{0!r} is not registered".format(fileobj))
KeyError: '<socket._socketobject object at 0x7f84e19e7de0> is not registered'

For completeness: I initially reported this on Google App Engine's issue tracker: GoogleCloudPlatform/python-compat-runtime#94

@benoitc benoitc added a commit that referenced this issue May 20, 2016
@benoitc when it happend there are good chance the socket has been removed bec…
…ause it timeouted on the other end. So ignore it.

fix #1258
98a2a65
@benoitc benoitc added a commit that referenced this issue May 20, 2016
@benoitc gthread: handle removed socket in the select loop
when it happend there are good chance the socket has been removed because it timeouted on the other end. So ignore it.

fix #1258
44d3ac8
@benoitc benoitc closed this in #1277 May 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment