Use gunicorn instead uvicorn in main

[bkocis]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import uvicorn

os.environ["TOKENIZERS_PARALLELISM"] = "false"


if __name__ == "__main__":
    uvicorn.run("start:app",
                host="0.0.0.0",
                port=8080,
                workers=2,
                limit_concurrency=70,
                backlog=300
                )

Description

I have an app using uvicorn and I have problems with accumulating memory usage. In addition, inside the app itself i use multiprocessing.Process to execute model inference using tensorflow in parallel processes.
I have tried many configurations and settings on uvicorn but the memory leak issue persists. At the moment I think that this is due to the thread handling done by uvicorn. I got a suggestion to use gunicorn instead of uvocorn, but I am not sure how I could simply switch from one to another, programmatically. The application is working, and returning the requests, there are no error messages. In the documentation the examples use uvicorn - can gunicorn be used just the same way as uvicorn in the examples?
My question is: would a switch from uvicorn to gunicorn make sense, and if so, would that impact the fastapi application?

Operating System

Linux

Operating System Details

Ubuntu 20.04
python3.7

pip packages:
fastapi==0.61.1
fastapi-route-logger-middleware==0.1.3
grpcio==1.32.0
joblib==0.17.0
numpy==1.18.5
optional.py==1.1.0
pandas==0.25.3
prometheus-fastapi-instrumentator==5.7.1
pydantic==1.7.1
python-multipart==0.0.5
tensorflow-gpu==2.3.2
uvicorn==0.12.2

FastAPI Version

fastapi==0.61.1

Python Version

Python 3.7.12

Additional Context

The application is used for implementing prediction using 3 tensorflow models. Two of which are run inside a multiproessing.process call, to achieve parallel execution.

[Kludex]

At the moment I think that this is due to the thread handling done by uvicorn.

The above doesn't makes sense.

What you may have heard is to use Gunicorn with Uvicorn workers, which shouldn't be able to solve your issue.

import uvicorn

os.environ["TOKENIZERS_PARALLELISM"] = "false"


if __name__ == "__main__":
    uvicorn.run("start:app",
                host="0.0.0.0",
                port=8080,
                workers=2,
                limit_concurrency=70,
                backlog=300
                )

Also, uvicorn doesn't even have "limit_concurrency" and "backlog" parameters. 😅

[bkocis]

Also, uvicorn doesn't even have "limit_concurrency" and "backlog" parameters. sweat_smile

uvicorn --help
--limit-concurrency INTEGER Maximum number of concurrent connections or
tasks to allow, before issuing HTTP 503
responses.
--backlog INTEGER Maximum number of connections to hold in
backlog

[Kludex]

Ups! My bad! Looks like I don't know uvicorn well enough. 😢

[bkocis]

No problem @Kludex - you are not alone, I thought setting up uvicorn will be easier....
Actually the very related issue to mine is Kludex/uvicorn#517

[Kludex]

Yeah... I'm going to talk to the maintainers to see if they can improve this on uvicorn. 😓

[ArcLightSlavik]

I don't why you would use uvicorn directly.

From uvicorn docs:

Deployment
Server deployment is a complex area, that will depend on what kind of service you're deploying Uvicorn onto.

As a general rule, you probably want to:

Run uvicorn --reload from the command line for local development.
Run gunicorn -k uvicorn.workers.UvicornWorker for production.

I usually end up running my development service behind gunicorn (to be consistent with production).
@Kludex Has something change so the production part isn't true anymore?

[Kludex]

I usually end up running my development service behind gunicorn (to be consistent with production).

The reload doesn't work with gunicorn, how do you do it? Is it different than this? Kludex/uvicorn#1193

@Kludex Has something change so the production part isn't true anymore?

Nothing changed, is just that now I understand more the reason why is the recommendation. Gunicorn is in charge of keeping the workers alive, killing them when needed, and spawning new ones as that happens. But if your setup already has a technology that guarantees that, then it's not needed.

That being said, on this post I think I made the wrong assumption, and it is needed. 😗

In any case, I'll be working on the "process manager" feature on uvicorn.

[ArcLightSlavik]

@Kludex >The reload doesn't work with gunicorn, how do you do it? Is it different than this?
I just write tests and have that running until I'm pretty confident with the endpoint (or whatever else) and then do a quick check in /docs or GUI.

[Kludex]

Work without hot reload is not convenient...

[Kludex]

For those arriving later, Sebastián just updated the docs: https://fastapi.tiangolo.com/deployment/server-workers/