New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gevent BlockingSwitchOutError sometimes occurs when sending signal USR1 #1645

Closed
ant1g opened this Issue Nov 17, 2017 · 6 comments

Comments

Projects
None yet
4 participants
@ant1g
Contributor

ant1g commented Nov 17, 2017

I am encountering the following error from time to time, when sending the USR1 signal to all gunicorn processes (to rotate log files).
Python 2.6
Gunicorn 19.7.1
Gevent 1.1.2

I found this old issue that could potentially be related:
#1044

Traceback

Traceback (most recent call last):
  File "/usr/lib/python/gunicorn-19.7.1-py2.6.egg/gunicorn/workers/base.py", line 177, in handle_usr1
    self.log.reopen_files()
  File "/usr/lib/python/gunicorn-19.7.1-py2.6.egg/gunicorn/glogging.py", line 351, in reopen_files
    handler.acquire()
  File "/usr/lib64/python2.6/logging/__init__.py", line 636, in acquire
    self.lock.acquire()
  File "/usr/lib64/python2.6/threading.py", line 123, in acquire
    rc = self.__block.acquire(blocking)
  File "/usr/lib/python/gunicorn-19.7.1-py2.6.egg/gunicorn/workers/base.py", line 177, in handle_usr1
    self.log.reopen_files()
  File "/usr/lib/python/gunicorn-19.7.1-py2.6.egg/gunicorn/glogging.py", line 351, in reopen_files
    handler.acquire()
  File "/usr/lib64/python2.6/logging/__init__.py", line 636, in acquire
    self.lock.acquire()
  File "/usr/lib64/python2.6/threading.py", line 123, in acquire
    rc = self.__block.acquire(blocking)
  File "gevent/_semaphore.py", line 198, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:4117)
  File "gevent/_semaphore.py", line 226, in gevent._semaphore.Semaphore.acquire (gevent/gevent._semaphore.c:3944)
  File "gevent/_semaphore.py", line 166, in gevent._semaphore.Semaphore._do_wait (gevent/gevent._semaphore.c:3178)
  File "/usr/lib/python/gevent-1.1.2-py2.6-linux-x86_64.egg/gevent/hub.py", line 608, in switch
    switch_out()
  File "/usr/lib/python/gevent-1.1.2-py2.6-linux-x86_64.egg/gevent/hub.py", line 612, in switch_out
    raise BlockingSwitchOutError('Impossible to call blocking function in the event loop callback')
BlockingSwitchOutError: Impossible to call blocking function in the event loop callback

Thanks!

@ant1g

This comment has been minimized.

Contributor

ant1g commented Nov 18, 2017

I monkey patched the GeventWorker class to override the handle_usr1 method and applied the same fix as proposed here #1044, spawning a new greenlet. I no longer see the error.
I will submit a PR.
Cheers

@ant1g

This comment has been minimized.

Contributor

ant1g commented Nov 27, 2017

Here #1651

@benoitc

This comment has been minimized.

Owner

benoitc commented Nov 28, 2017

hrm apparently it blocks while trying to log. did anyone figued why?

@ant1g

This comment has been minimized.

Contributor

ant1g commented Nov 28, 2017

All workers do this upon receiving the USR1 signal:
def handle_usr1(self, sig, frame): self.log.reopen_files()

The reopen_files() function in glogging will then do this:
with self.lock:...

This is most likely the culprit!

@tilgovi

This comment has been minimized.

Collaborator

tilgovi commented Dec 4, 2017

In any case, doing any significant work on a signal handler is a bad idea to me. Reopening log files could mean lots of things, depending on the logging implementation and filesystem. It's probably best to merge this at #1651.

@berkerpeksag

This comment has been minimized.

Collaborator

berkerpeksag commented Jan 25, 2018

PR #1651 has been merged. Closing this as 'fixed'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment