New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

killling workers takes too much time #844

Open
ryanking8215 opened this Issue Feb 16, 2015 · 13 comments

Comments

Projects
None yet
9 participants
@ryanking8215

ryanking8215 commented Feb 16, 2015

My environment is Flask app and a user thread to listen the redis's message.
I used uwsgi to drive the app.
But when I press ctrl-c to quit the uwsgi, uwsgi will take long time and say:

workers … is taking too much time to die…NO MERCY

If I disable the user thread , everything is ok.

I tried many ways to resolve it.

signal
uwsgi.signal
uwsgi.atexit

But they all don't work.

here is my uwsgi command

uwsgi --http-socket 127.0.0.1:8800 --virtualenv=/opt/python_venv --wsgi-file /opt/app.py --callable app --processes 4 --threads 2 --stats 127.0.0.1:9191

Thanks for any suggestion!

@unbit

This comment has been minimized.

Owner

unbit commented Feb 16, 2015

Destroying threads in the POSIX world is basically no-way :) The right approach is letting your thread return when the destroy procedure is triggered. You have a bunch of ways to do it, but personally i am way more brutal and i generally set --worker-reload-mercy to a couple of seconds :)

@ryanking8215

This comment has been minimized.

ryanking8215 commented Feb 16, 2015

In fact, I use uwsgi.atexit to get the destroy signal successfully , and in the callback I call the function which make redis listening thread quit. But worker still can't be killed until timeout.

@unbit

This comment has been minimized.

Owner

unbit commented Feb 18, 2015

How do you 'quit' the thread ?

@ryanking8215

This comment has been minimized.

ryanking8215 commented Feb 19, 2015

I wrote a simple demo, the problem is I couldn't get the destroy signal.

from flask import Flask
import threading
import time

app=Flask(__name__)

@app.route('/')
def index():
    return 'hello,world'

a=1
def myjob():
    print("my job run",a)
    while a!=0:
        time.sleep(1)
    print(">>>>>>>> my job quit")

thrd = threading.Thread(target=myjob)
thrd.start()

def bye():
    print(">>>>>>>>>>> bye")
    a=0

try:
    import uwsgi
    uwsgi.atexit = bye
except:
    pass

import signal
signal.signal(signal.SIGINT,bye)
signal.signal(signal.SIGQUIT,bye)

if __name__=='__main__':
    app.run()

Neither uwsgi.atext nor signal could get the destroy signal. Unless the thread is not started.

@anthonyrisinger

This comment has been minimized.

Contributor

anthonyrisinger commented Mar 4, 2015

You did not enable app threads with --enable-threads, probably you are having GIL issues.

Also, you cannot catch signals in python without a specific option enabled that I forget offhand, possibly only available in 2.1.

In general, you should avoid --threads for python apps... they cannot be executed concurrently because of GIL. That option creates C-level threads (unmanaged by python) to run your WSGI app in.

edit: also, your bye function will not work because a is a global; you need to add global a to the top of the function.

@ryanking8215

This comment has been minimized.

ryanking8215 commented Mar 13, 2015

You are right about global a stuff. -:)
I know the threads is not the proper way in python because of GIL. -:)

But the problem is when I pressed 'Ctrl-c' to quit the uwsgi, bye() function was not called. I don't know how to make it called to quit the thread

@xrmx

This comment has been minimized.

Collaborator

xrmx commented Nov 10, 2015

@ryankask is this still an issue? btw if you are doing pubsub the redis-py's pubsub object has a run_in_thread method to help you with this.

@Vingtoft

This comment has been minimized.

Vingtoft commented Feb 16, 2016

Did anyone find a solution to this problem? I have the exact same issue.

@Kronuz

This comment has been minimized.

Kronuz commented Jun 28, 2016

Same problem here... uwsgi.atexit didn't work and neither using signal.signal nor uwsgi.signal Is there a bug or this just can't be done using uwsgi.

@unbit

This comment has been minimized.

Owner

unbit commented Jun 28, 2016

First of all, destroying threads is not possible on POSIX systems without thread cooperation itself, unfortunately there is no easy-solution. If you want to rely on UNIX signals (and very probably you are opening a new can of worms, but they should at least work 99% of the times) you have to explicitely tell the python VM you want to override signals management using the --py-call-osafterfork option.

@benjolitz

This comment has been minimized.

benjolitz commented Jan 5, 2017

I've learned today that if my flask app on uWSGI uses enable-threads=true and threads=2 (for any number greater than 1), then a SIGHUP on a worker produces an indefinite hang.

@zsluedem

This comment has been minimized.

zsluedem commented Sep 8, 2017

i meet this kind of issue too.My config threads=2.
When i try to kill the uwsgi byuwsgi --stop uwsgi.pid, it wait for a while to kill and show the log take too much time.
And when i try to realod the uwsgi , the uwsgi can't accept any request any more. It just hang there and do nothing.But sometimes it worked, it is kind of mysterious.
My app is a python app.And i found this and it seems python doesn't support threaded.

I will try to use the app without threaded for a while and see how.

@erny

This comment has been minimized.

erny commented Sep 29, 2017

Could this be similar to my issue #1599 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment