Scale FastApi with sync endpoints

[dapollak]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import logging
from fastapi import FastAPI

app = FastAPI()
logger = logging.getLogger()

@app.get("/")
def root():
    logger.info(f"Running on {os.getpid()}")
    time.sleep(3600)
    return {"message": "Hello World"}

Description

I've noticed lately that we have some latency problems when the servers are busy. I dived into it and found out that, if for example, I have four uvicorn workers, while one worker is very busy, the rest three are significantly less busier. That has two problems -

1. We're not taking advantage of all our parallelism power
2. We're suffering more from Python's GIL problems on the same worker.

In the example code, if I run it with gunicorn main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000, the first five requests will result in the following output -

INFO:root:Running on 643
INFO:root:Running on 643
INFO:root:Running on 643
INFO:root:Running on 643
INFO:root:Running on 642

I investigated uvicorn workers and found out they use asyncio server interface, which accepts new connections on python's event loop, which means that as long we're using not-async endpoints (which run on AnyIo threads), there is no limit on the number of new connections a worker gets, which again takes us back to the GIL problem.

To sum it all up, I'm a little confused about what is my best option for using FastApi with not-async endpoints and get better performance. Should I use more workers? Should I use more container instances?
Maybe we should implement a new worker which able to control better multithreading on such cases ? I have some implementation ideas for this purpose, but I’d like to have your advice here.

Operating System

Linux

Operating System Details

FastAPI Version

0.88.0

Python Version

3.9.4

Additional Context

[extreme4all]

Notes for the reader:

• Concurrency is when 2 or more tasks are being performed at the same time, which might mean that only 1 of them is being worked on while the other ones are paused.

• Parallelism is when 2 or more tasks are executing at the same time.

Concurrency:

• gunicorn: --workers, spawn different processes
• gunicorn: --threads, per worker spawn threads, threads share memory in the same worker
• Async: pseudo threads, asyncio uses one thread & one process, and allows concurrency by jumping between coroutines (async functions)

Parallelism:

• gunicorn: --workers
• gunicorn: --threads

In your example you are using the workers, and the issue that is occurring is that the work is not balanced of the workers.

According to the documentation:

The default synchronous workers assume that your application is resource-bound in terms of CPU and network bandwidth.

And:

Gunicorn relies on the operating system to provide all of the load balancing when handling requests.

if an application is CPU bound ("resource bound") than workers are ideal, if an application is I/O bound threads or pseudo threads are ideal.

[extreme4all]

• is there any reason why async is not an option?

[dapollak]

@extreme4all because we're using not asyncio compatible library (a snowflake driver of sqlalchemy

[jgould22]

In addition to what @extreme4all posted

I investigated uvicorn workers and found out they use asyncio server interface, which accepts new connections on python's event loop

This is not always true, Unicorn as the name implies will use uvloop depending on how it is configured / installed

Another thing to consider is that FastAPI/Starlette are spawning those def endpoints onto a threadpool, depending on its implementation that could be where you are seeing a bottle neck.

Pythons threads/process story can be difficult to reason about.

In the short term depending on the machine you have deployed this on Gunicorn will recommend 2 workers per core. However if the cores are not busy you can probably up the worker count and that may solve your problem / give you some breathing room while you investigate further.

[dapollak]

@jgould22 I know that def endpoints are spawned with threadpool threads, that's why I suspect we experience low latencies due to GIL limitations with many threads on the same worker (the amount of threads is not limited on each worker)

[iudeen]

The best approach for a scalable deployment (in containerized environments) is using barebones Uvicorn with single worker and scale your containers horizontally.

You can have health check APIs to ensure your containers don't go stale.

[talboren]

The best approach for a scalable deployment (in containerized environments) is using barebones Uvicorn with single worker and scale your containers horizontally.

You can have health check APIs to ensure your containers don't go stale.

@iudeen I'd love hearing where did you find this to be the best approach

[iudeen]

@talboren based on hands-on experience on a product that has hundreds of thousands concurrent users and in a way from FastAPI docs as well: https://fastapi.tiangolo.com/deployment/docker/#replication-number-of-processes

[Kludex]

there is no limit on the number of new connections a worker gets

This is not true.

The question was already answered on the first comment:

"Gunicorn relies on the operating system to provide all of the load balancing when handling requests."

This is on gunicorn.

Ref.: https://docs.gunicorn.org/en/latest/design.html?highlight=operating%20system#how-many-workers

[dapollak]

@Kludex The uvicorn workers don't limit the threads that the fastapi app creates, when using def endpoints. As I stated at the beginning of the thread, when a new connection is accepted, it's passed to the fastapi app which creates a new thread using AnyIO library, and then the event loop of the uvicorn worker gets available right away to accept new connections.

[Kludex]

Where does the uvicorn worker limits the threads?

Also, I didn't get the point of the previous message. Can you rephrase it?

[dapollak]

Where does the uvicorn worker limits the threads?

Also, I didn't get the point of the previous message. Can you rephrase it?

It doesn't limit, forgot a word 😅
The point is that the combination of unbalanced load on the worker and "unlimited" threads per is bad due to both GIL problems and not using all workers

[Kludex]

Uvicorn doesn't limit, but AnyIO does. There's a limit of 40 threads by default.

...but the above is not relevant to this discussion.

Gunicorn is in charge of the load balancing, as mentioned above - and this is the relevant part here.

[dapollak]

But gunicorn doesn't do it. It's documented -

Gunicorn relies on the operating system to provide all of the load balancing when handling requests.

So, with def endpoints, threads creation allows the same worker to get all the connection, so I suspect that this is what causes to the the workers being unbalanced