Socket closing error

[prostmich]

In my project I use gunicorn with uvicorn worker class.
The gunicorn process is bound to the Unix socket and after sending the SIGHUP signal, the following error message is displayed in the log:

[INFO] Starting gunicorn 20.1.0
[INFO] Listening at: http://127.0.0.1:8001 (30650)
[INFO] Using worker: uvicorn.workers.UvicornWorker
[INFO] Booting worker with pid: 30652
[INFO] Started server process [30652]
[INFO] Waiting for application startup.
[INFO] Application startup complete.
[INFO] Handling signal: hup
[INFO] Hang up: Master
[INFO] Booting worker with pid: 30689
[INFO] Shutting down
[INFO] Error while closing socket [Errno 9] Bad file descriptor <<<--- 
[INFO] Waiting for application shutdown.
[INFO] Application shutdown complete.
[INFO] Finished server process [30652]
[INFO] Worker exiting (pid: 30652)
[INFO] Started server process [30689]
[INFO] Waiting for application startup.
[INFO] Application startup complete.

Minimal reproducible example:

Requirements:

uvicorn==0.20.0
gunicorn==20.1.0
fastapi==0.88.0

Launch:

gunicorn --worker-class "uvicorn.workers.UvicornWorker" "main:app"

main.py

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def root():
    return "Hello world"

Send HUP signal:

kill -s HUP {gunicorn master pid}

Possible solution:

It seems to be a bug in the socket closing sequence and I may have a solution.
We should close all sockets first and then all asyncio servers. It looks like the server has already closed the socket connection and we are trying to close it again in the next for loop.

server.py:

async def shutdown(self, sockets: Optional[List[socket.socket]] = None) -> None:
    logger.info("Shutting down")

    # Stop accepting new connections.
    for server in self.servers:
        server.close()
    for sock in sockets or []:
        sock.close()
    for server in self.servers:
        await server.wait_closed()

    # Request shutdown on all existing connections.
    for connection in list(self.server_state.connections):
        connection.shutdown()
    await asyncio.sleep(0.1)

fixed server.py:

async def shutdown(self, sockets: Optional[List[socket.socket]] = None) -> None:
    logger.info("Shutting down")

    # Stop accepting new connections.
    for sock in sockets or []:
        sock.close()
    for server in self.servers:
        server.close()
    for server in self.servers:
        await server.wait_closed()

    # Request shutdown on all existing connections.
    for connection in list(self.server_state.connections):
        connection.shutdown()
    await asyncio.sleep(0.1)

I know it may seem like a small thing, but it would get rid of the misunderstanding.

[Kludex]

This was intentionally introduced to support "shutdown" events using gunicorn.

You can read more about it here: #1710.

I've made gunicorn aware of my intentions: benoitc/gunicorn#1877 (comment).

[prostmich]

Thanks, I saw this thread, but the error message in the logs is not expected behavior, right?
And we can easily fix that right now

[Kludex]

Thanks, I saw this thread, but the error message in the logs is not expected behavior, right?

Well, it actually was...

And we can easily fix that right now

Changing the order of sock.close and server.close is not a solution, based on what I remember. Because in the past it was actually like this, and it was changed to the current behavior to solve another bug. Also, we shouldn't change the uvicorn core logic to satisfy gunicorn. That said... If you want to help, you can confirm what I'm saying (it really goes back some years ago, and this logic was actually introduced on a different file). Besides that, you might want to check if we really need to run sock.close, because in theory the server.close is already doing that.