You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The bug is reproducible against the latest release and/or master.
There are no similar issues or pull requests to fix it yet.
Describe the bug
When starting Gunicorn with Uvicorn worker(s), if the app uses subprocess to start other processes and captures the output, their returncode is in most cases 0, even if the actual exit code was 1.
To reproduce
Take this minimal FastAPI app (or replace with Starlette), main.py:
Open the browser at http:127.0.0.1:8000/docs and send a request to /run.
Expected behavior
The detected returncode should always be 1, as the subprocess always exits with 1.
Actual behavior
In most of the cases it will return a returncode of 0. Strangely enough, in some cases, it will return a returncode of 1.
Debugging material
This is because the UvicornWorker, which inherits from the base Gunicorn worker, declares a method init_signals() (overriding the parent method) but doesn't do anything. I suspect it's because the signal handlers are declared in the Server.install_signal_handlers() with compatibility with asyncio.
But the UvicornWorker process is started with os.fork() by Gunicorn (if I understand correctly) and by the point it is forked, the Gunicorn "Arbiter" class (that handles worker processes) already set its own signal handlers.
And the signal handlers in the Gunicorn base worker reset those handlers, but the UvicornWorker doesn't. So, when a process started with subprocessing is terminated, the SIGCHLD signal is handled by the Gunicorn Arbiter (as if the terminated process was a worker) instead of by the UvicornWorker.
Disclaimer: why the SIGCHLD signal handling in the Gunicorn Arbiter alters the returncode of a process run with subprocess, when capturing output, is still a mystery to me. But I realized the signal handler in the Arbiter is expected to handle dead worker processes. And worker subclasses all seem to reset the signal handlers to revert those signals set by the Arbiter.
I'm also submitting a PR to fix this: #895. It's just 3 lines of code. But debugging it and finding it took me almost a week. 😅
Environment
OS / Python / Uvicorn version: just run uvicorn --version: Running uvicorn 0.13.1 with CPython 3.8.5 on Linux (it's actually installed from source, for debugging)
Gunicorn version (also installed from source, for debugging): gunicorn (version 20.0.4)
The exact command you're running uvicorn with, all flags you passed included. If you run it with gunicorn please do the same. If there is a reverse-proxy involved and you cannot reproduce without it please give the minimal config of it to reproduce.
Checklist
master
.Describe the bug
When starting Gunicorn with Uvicorn worker(s), if the app uses
subprocess
to start other processes and captures the output, theirreturncode
is in most cases0
, even if the actual exit code was1
.To reproduce
Take this minimal FastAPI app (or replace with Starlette),
main.py
:Then run it with:
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Open the browser at http:127.0.0.1:8000/docs and send a request to
/run
.Expected behavior
The detected
returncode
should always be1
, as the subprocess always exits with1
.Actual behavior
In most of the cases it will return a
returncode
of0
. Strangely enough, in some cases, it will return areturncode
of1
.Debugging material
This is because the
UvicornWorker
, which inherits from the base Gunicorn worker, declares a methodinit_signals()
(overriding the parent method) but doesn't do anything. I suspect it's because the signal handlers are declared in theServer.install_signal_handlers()
with compatibility withasyncio
.But the
UvicornWorker
process is started withos.fork()
by Gunicorn (if I understand correctly) and by the point it is forked, the Gunicorn "Arbiter" class (that handles worker processes) already set its own signal handlers.And the signal handlers in the Gunicorn base worker reset those handlers, but the
UvicornWorker
doesn't. So, when a process started withsubprocessing
is terminated, theSIGCHLD
signal is handled by the GunicornArbiter
(as if the terminated process was a worker) instead of by theUvicornWorker
.Disclaimer: why the
SIGCHLD
signal handling in the GunicornArbiter
alters thereturncode
of a process run withsubprocess
, when capturing output, is still a mystery to me. But I realized the signal handler in theArbiter
is expected to handle dead worker processes. And worker subclasses all seem to reset the signal handlers to revert those signals set by theArbiter
.I'm also submitting a PR to fix this: #895. It's just 3 lines of code. But debugging it and finding it took me almost a week. 😅
Environment
uvicorn --version
:Running uvicorn 0.13.1 with CPython 3.8.5 on Linux
(it's actually installed from source, for debugging)gunicorn (version 20.0.4)
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Additional context
I'm pretty sure this issue #584 is related to the same problem.
The text was updated successfully, but these errors were encountered: