Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uWGSI graceful shutdown #849

Open
dmk23 opened this Issue Feb 23, 2015 · 27 comments

Comments

Projects
None yet
9 participants
@dmk23
Copy link

commented Feb 23, 2015

I am looking at the full list of uWGSI signals to figure out the proper way to shut down:
http://uwsgi-docs.readthedocs.org/en/latest/Management.html

It is quite surprising to see no way to exit the service gracefully vs. just brutally. Yes, it is good to be able to gracefully reload, but some deployments may rely on outside scripts invoking a "service restart" command, which would stop and then start the service - in this case destroying the requests in process.

Is there a workaround to enforce graceful shutdown? A possible new feature? Why have two signals (SIGINT and SIGQUIT) do exact same thing when one could designate a graceful option?

@unbit

This comment has been minimized.

Copy link
Owner

commented Feb 23, 2015

check the master fifo. it is the way to go in modern environments

@dmk23

This comment has been minimized.

Copy link
Author

commented Feb 23, 2015

OK, found it already.

One suggestion though - why not add config options for overriding any signal? It just seems quite cumbersome to have to set up these pipes for something as simple as graceful shutdown...

@unbit

This comment has been minimized.

Copy link
Owner

commented Feb 24, 2015

Well, the signal+pidfile paradigm is so unreliable that people spit out new solutions every time by ages, so solving it with a fifo (that is automatically created by uWSGI) seemed a good compromise :)

By the way, just make a proposal, if it is easy (and not invasive) to implement we can absolutely add it

@dmk23

This comment has been minimized.

Copy link
Author

commented Feb 24, 2015

Definitely did not mean to criticize - been using uWSGI for just a few days as a replacement of a very old and unreliable container and very impressed with what you've done here :)

Do you want a proposal in any specific format? It is really simple, a config like this:

signal-command = SIGQUIT r

This would override the "default" behavior of SIGQUIT to be identical to 'r' sent to Master FIFO, effecting a graceful shutdown. The benefit is not having to set up Master FIFO and track / manage / access these files if we only care about a couple simple commands.

It should be possible to have multiple lines like this to override multiple signals.

Does this sound clear/easy enough to implement?

@unbit

This comment has been minimized.

Copy link
Owner

commented Feb 24, 2015

So you suggest to directly maps 'fifo' commands to unix signals. It could be an interesting approach

@dmk23

This comment has been minimized.

Copy link
Author

commented Feb 24, 2015

Yes - think of this as a different interface to submit the same commands...

@WST

This comment has been minimized.

Copy link

commented Feb 24, 2015

I confirm this could be a nice feature to have (if it’s easy to add).

@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2015

@dmk23, AFAIK, sending SIGINT to master process is a graceful shutdown; uWSGI waits for the workers to complete their request (for a configurable amount of time, grep --help for word mercy) before brutally killing them.

@unbit is talking about the unreliability of pidfiles in general, which is what the "convenience" tools rely on.

@dmk23

This comment has been minimized.

Copy link
Author

commented Mar 4, 2015

I've confirmed that the shutdown on SIGINT is brutal and merciful one is not possible with any current signals... Only reload by signal can be merciful... Note the page linked in the OP.

@unbit

This comment has been minimized.

Copy link
Owner

commented Mar 4, 2015

@xtfxme the proposal made by @dmk23 is interesting (mapping signals to fifo commands), someone forced me to note that some filesystem (in the embedded world) does not support named pipes (nor unix sockets), so signals are the only viable approach (albeit suboptimal).

@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2015

What's the purpose of --reload-mercy and friends (which mention reload or shutdown) if SIGINT was never clean? I swear I tested this years ago and the callable was allowed to finish, but apparently I was mistaken, so this is very much news to me :-(

I'm trying now to recall situations where uWSGI is forced to brutally kill the worker... even in a busy loop with --py-reload-osafterfork (which I expected a KeyboardException to be thrown in Python) I can't seem to shutdown properly.

@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2015

I'm just amazed I never noticed this :-) all this time I was thinking the INT and HUP code paths were the same as I haven't worked on or walked those paths myself, with the only real difference being HUP continued... the good news is our apps have tolerated it just fine for a very long time!

@unbit

This comment has been minimized.

Copy link
Owner

commented Mar 4, 2015

The SIGINT triggers other SIGINT (a signal that workers are allowed to trap) and wait for processes death. It could not happen (maybe for a stuck worker, or an overriden signal). That is why mercy is required (in addition to graceful reloads). So technically SIGINT is not very brutal, this is why you probably never noticed it

@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2015

The mapping of OS sigs to management chars does seem interesting, but the OP, and I suppose others, just want to be able to kill the stack without interrupting any active requests -- couldn't we do something like (pseudo):

def INT_handler():
    return HUP_handler(restart=False)

def HUP_handler(restart=True):
    ...
    if restart:
        execv(...)
    else:
        exit(0)

To me, a graceful shutdown is little more than a graceful restart that never actually restarts. Would there be problems with listen queue filling during this time, and then they are lost after the process dies? Or what am I missing here?

@dmk23

This comment has been minimized.

Copy link
Author

commented Mar 4, 2015

I suppose uWSGI could respond to the remaining requests in the queue with 503, when shutting down.

@NicolasLM

This comment has been minimized.

Copy link

commented Jul 4, 2015

Any news on this issue? I still couldn't find a way to shutdown uwsgi without killing the ongoing requests.

@unbit

This comment has been minimized.

Copy link
Owner

commented Jul 4, 2015

@NicolasLM this is about mapping UNIX signals to master-fifo commands for those setups where fifos are not available. You can safely graceful shutdown using them.

http://uwsgi-docs.readthedocs.org/en/latest/MasterFIFO.html

@NicolasLM

This comment has been minimized.

Copy link

commented Jul 6, 2015

Looking at the base issue, it seems pretty close to what I'm trying to achieve: running uwsgi as PID 1 in a Docker container. When Docker stops a container it sends a SIGTERM to the process with PID 1, here the uwsgi master.

There is the --die-on-term flag in uwsgi that allows to kill workers instead of reloading them when receiving a SIGTERM, why not adding --stop-on-term to let the workers finish their job?

I understand that pipes are way more flexible than a bunch of signals but in some environments signals are easier.

@unbit

This comment has been minimized.

Copy link
Owner

commented Jul 6, 2015

In master and uwsgi-2.0 branch there is a new hook action called "unix_signal" that allows you to remap any unix signal number to a specific function symbol:

./uwsgi --hook-master-start "unix_signal:1 gracefully_kill_them_all" --http-socket :9090 -M

will call gracefully_kill_them_all() every time signal 1 is received

This is obviously a way more generic solution to the problem described in this issue

@NicolasLM

This comment has been minimized.

Copy link

commented Jul 6, 2015

Great, it works! This a pretty flexible approach. Thanks.

@ionelmc

This comment has been minimized.

Copy link

commented Jul 7, 2015

Is there a list of all the available function symbols? Can those functions be user-defined?

@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Jul 7, 2015

Looks like anything in the process space

@unbit

This comment has been minimized.

Copy link
Owner

commented Jul 7, 2015

@ionelmc yes anything in the process address space, so eventually even a symbol exposed by a library

#include <stdio.h>

void hello_world(int signum) {
        printf("Hello World\n");
}
gcc -o libhello.so -shared hello.c
uwsgi --dlopen ./libhello.so --hook-master-start "unix_signal:1 hello_world" ...
@anthonyrisinger

This comment has been minimized.

Copy link
Contributor

commented Jul 7, 2015

Does this allow signal handlers set by workers (python) a chance to respond? Atexit, etc?

What happens to backlogged connections during shutdown (in general)?

If the worker misbehaves, master still comes in after the time out with a brutal kill, yes? If things work well, will this be default?

Maybe interesting to note, embed_config would allow for a uWSGI build with this behavior, by default, today :-)

@xrmx xrmx added the documentation label Nov 13, 2015

@askoretskiy

This comment has been minimized.

Copy link

commented Aug 2, 2016

Is it possible that somebody adds this to uwsgi documention? Maybe another section "Shutdown" with these recipes -- brutal shutdown and graceful one.

At the moment it is rather hard to find these recipes.

@braedon

This comment has been minimized.

Copy link

commented Sep 18, 2017

Sorry to bump this, but the solution described here still appears to be undocumented.

I also can't find an answer to this question (asked by @anthonyrisinger above):

What happens to backlogged connections during shutdown (in general)?

@unbit

This comment has been minimized.

Copy link
Owner

commented Sep 18, 2017

@braedon pull-requests for docs are way more welcomed than code one, so if anyone want to write it i will merge it for sure.

Regarding backlogged connections RST should be generated as no socket is no more bound to that address/port pair

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.