Production configuration of gunicorn

[sandys]

NOTE: I'm not sure whether i should be filing this here or upstream on starlette. Please let me know if Starlette developers are more knowledgeable about this. However, I'm using fastapi.

What is a good configuration setting for gunicorn in production.

On reading documentation of gunicorn with uvicorn, there are lines like this

"Gunicorn provides a different set of configuration options to Uvicorn, so some options such as --limit-concurrency are not yet supported when running with Gunicorn"

Is it possible for you to share a list of options that work well ? One of the biggest question is around threads - should we set the threads option?
what about stuff like "worker_connections" , "keepalive" (especially in context of async code)

[dmontagu]

This might be a reasonable starting point: https://github.com/tiangolo/uvicorn-gunicorn-docker/blob/master/python3.7/gunicorn_conf.py

But it doesn't go into much depth on the available options.

The starlette and uvicorn repos would also be good places to ask this question / search for related issues.

[devsetgo]

I started mine off from a gunicorn example https://github.com/benoitc/gunicorn/blob/master/examples/example_config.py

Then set mine like this - https://github.com/devsetgo/test-api/blob/master/app/gunicorn_cfg.py

From my experience, i have found that I really just need to set workers, bind, port, and for starlette/fastapi the worker class is what makes uvicorn work with gunicorn.

[sandys]

Hi mike Are you setting threads in gunicorn ? Im not sure if the Uvicorn worker plays well with threads. What's the things you tune for in production?

…

On Sun, 22 Sep, 2019, 05:29 Mike Ryan, ***@***.***> wrote: I started mine off from a gunicorn example https://github.com/benoitc/gunicorn/blob/master/examples/example_config.py Then set mine like this - https://github.com/devsetgo/test-api/blob/master/app/gunicorn_cfg.py From my experience, i have found that I really just need to set workers, bind, port, and for starlette/fastapi the worker class is what makes uvicorn work with gunicorn. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU2O5KBYPNMLAPO63UDQK2YONA5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7I3T4Q#issuecomment-533838322>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU67APO7ZOC4ELT2LWDQK2YONANCNFSM4IYM5USQ> .

[devsetgo]

The item I am most concerned with for my production use are the number of workers. For async I've tried a couple different worker classes, but didn't see much change. So I just use uvicorn.workers.UvicornWorker. I set the backlog of connections, but I haven't ever found it to be an issue.

In my opinion, I don't think threads will matter. Threads I presume will matter more if you are kicking on new threads using multi-threading. I've not tried threads with Gunicorn for sync or async, so that could be an incorrect assessment.

I've done some testing of my test-api using WRK and connecting to my server 1000 miles away (1600 Km). So far with 9 workers running (4 cores x 2 + 1) I max out just below 6k requests per second. I find that Traefik and Let'sEncrypt drops that to 2k (research area). A single core (1 x 2 + 1) I can get around 2k requests per second (drops to 1k with Traefik/LetsEncrypt). I eventually plan to do more tests, but this is good enough for my use and horizontal scaling would increase this easily.

[sandys]

Thanks for that info. What tuning are you doing for backlog? Would love to know learnings here. P.S. we are using haproxy - we find the config to be simpler and much higher performance.

…

On Sun, 22 Sep, 2019, 23:04 Mike Ryan, ***@***.***> wrote: The item I am most concerned with for *my* production use are the number of workers. For async I've tried a couple different worker classes, but didn't see much change. So I just use *uvicorn.workers.UvicornWorker* . I set the backlog of connections, but I haven't ever found it to be an issue. In my opinion, I don't think threads will matter. Threads I presume will matter more if you are kicking on new threads using multi-threading. I've not tried threads with Gunicorn for sync or async, so that could be an incorrect assessment. I've done some testing of my test-api using WRK and connecting to my server 1000 miles away (1600 Km). So far with 9 workers running (4 cores x 2 + 1) I max out just below 6k requests per second. I find that Traefik and Let'sEncrypt drops that to 2k (research area). A single core (1 x 2 + 1) I can get around 2k requests per second (drops to 1k with Traefik/LetsEncrypt). I eventually plan to do more tests, but this is good enough for my use and horizontal scaling would increase this easily. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU4BZE6RGPLLJIFBG43QK6UA3A5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7JLAJA#issuecomment-533901348>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYUZLA7OCYKPTOCRBJBTQK6UA3ANCNFSM4IYM5USQ> .

[devsetgo]

Backlog setting came as a recommendation to add from a friend to add. The maximum is 2048 if I remember right, which is what I use as a setting. I've tried a few changes to the setting, but for my own use it has not made any difference.

Haven't used HAProxy, but I was using Nginx for the last 10 years and jwilder/nginx-proxy. Performance was pretty good (better than Traefik), but I find configuration is easier with Traefik + LetsEncrypt and Docker integration. So that is why I have been using it for the last 6 months.

Thanks for the tip on HAProxy, I'll look into it.

Edit - Love the name of your domain. Made me laugh... excellent choice.

[euri10]

I think most of your questions are answered quite well here @sandys https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7

[sandys]

Hi Thanks for replying . We already use gunicorn in a high load site with flask. We are very familiar with this. We also use uwsgi so probably have an entire gamut of understanding there. We do not understand gunicorn with Uvicorn workers. From what I understand, threads are recommended against. And the way toncreate pools+sessions is also very different. So the link you pointed out is not very helpful in this specific context.

…

On Wed, 25 Sep, 2019, 14:51 euri10, ***@***.***> wrote: I think most of your questions are answered quite well here @sandys <https://github.com/sandys> https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU7TBDNRJY7UE766YODQLMURHA5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RG7JY#issuecomment-534933415>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU2R625IGMR55V2TVWTQLMURHANCNFSM4IYM5USQ> .

[euri10]

So the link you pointed out is not very helpful in this specific context.

so sorry it didn't help @sandys , next time I'll google something more useful that will fit your exact needs, hopefully 😜

[sandys]

Oh I'm very thankful that you replied. It shows the help of the community and I thank you for that. Fastapi is very new and we are also trying to figure out production stuff

…

On Wed, 25 Sep, 2019, 18:16 euri10, ***@***.***> wrote: So the link you pointed out is not very helpful in this specific context. so sorry it didn't help @sandys <https://github.com/sandys> , next time I'll google something more useful that will fit your exact needs, hopefully 😜 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU5TQRFE7HHOWTBUIFTQLNMSTA5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RYNQA#issuecomment-535004864>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU4R2M6OQ5VYRLNSA7LQLNMSTANCNFSM4IYM5USQ> .

[dmontagu]

@sandys Thanks for your feedback/contributions! Please keep sharing as you figure out more of the production tuning stuff, I'm definitely hoping to gain some knowledge from these tuning efforts. (Sorry I can't be more helpful on this point...)

Have you tried asking in the uvicorn / starlette repos? I think there is a broader (and often very technically sophisticated) audience watching those repos.

[sandys]

yes i have asked - encode/starlette#642 but i havent gotten any replies yet.

…

On Wed, Sep 25, 2019 at 6:31 PM dmontagu ***@***.***> wrote: @sandys <https://github.com/sandys> Thanks for your feedback/contributions! Please keep sharing as you figure out more of the production tuning stuff, I'm definitely hoping to gain some knowledge from these tuning efforts. (Sorry I can't be more helpful on this point...) Have you tried asking in the uvicorn / starlette repos? I think there is a broader (and often more technically sophisticated) audience watching those repos. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU5S5K6P6G7RF4RJVOLQLNOJVA5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7RZ4FQ#issuecomment-535010838>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU5EM4IPCEHRB4ZH3XDQLNOJVANCNFSM4IYM5USQ> .

[krnr]

I've been trying to enable pre_request/post_request hooks for Uvicorn worker, but with no avail. They are supported, but why their behavior is different from standard worker?

pre_request set up as usual

loglevel = "debug"
worker_class = "uvicorn.workers.UvicornWorker"

def pre_request(worker, req):
    worker.log.warning('foo')

has no effect. but set as

def pre_request(worker):
    worker.log.warning('UUUU')

results in error: Invalid value for pre_request: <function pre_request at 0x7feb8dee0a60>. so, it's processed, but how? looking at the debug output:

  pre_request: <function pre_request at 0x7f9bb01f5a60>
  post_request: <function validate_post_request.<locals>.<lambda> at 0x7f9bb01f5b70>

why post_request is a lambda function and how to set it?

when I use standard sync worker everything is fine.

[tiangolo]

Thanks everyone for the discussion!

@sandys as I understand, threads in Gunicorn are useful for WSGI frameworks that can run in multiple threads, if supported by the rest of the components used. And that way increase concurrency.

With an ASGI framework, it's already handling concurrency on the async loop. In fact, Starlette and FastAPI will run some parts of the code (when you use normal def) in a thread pool.

So, I don't think enabling threads with Uvicorn would help (or maybe even harm).

---

@krnr that seems to be a separated problem, can you please create a new issue for that?

[sandys]

Hi Sebastian Thanks for replying. Threads here represent something entirely different than threading in fastapi. Gunicorn also allows for each of the workers to have multiple threads. In this case, the Python application is loaded once per worker, and each of the threads spawned by the same worker shares the same memory space. Each worker thread is a fastapi application and can give the full benefit of async. Atleast that's the theory. Again, I'm not second guessing what you are telling me...I'm asking for the optimal tuning for an extremely high volume API. There's documentation for other kinds of workers on gunicorn...but not Uvicorn/fastapi. That's why I'm requesting for some info by teams who have run this under high load.

…

On Sun, 27 Oct, 2019, 17:41 Sebastián Ramírez, ***@***.***> wrote: Thanks everyone for the discussion! @sandys <https://github.com/sandys> as I understand, threads in Gunicorn are useful for WSGI frameworks that can run in multiple threads, if supported by the rest of the components used. And that way increase concurrency. With an ASGI framework, it's already handling concurrency on the async loop. In fact, Starlette and FastAPI will run some parts of the code (when you use normal def) in a thread pool. So, I don't think enabling threads with Uvicorn would help (or maybe even harm). ------------------------------ @krnr <https://github.com/krnr> that seems to be a separated problem, can you please create a new issue for that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551?email_source=notifications&email_token=AAASYU3NEP5HKQC4EK45MK3QQWAQTA5CNFSM4IYM5US2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECK433Q#issuecomment-546688494>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASYUYLD4RPK7YQZ24WZYTQQWAQTANCNFSM4IYM5USQ> .

[dmontagu]

It might be better to ask questions like this on the gunicorn github than fastapi; I expect the same advice will apply to fastapi as applies to any async server framework.

This is a relatively old comment but I suspect little has changed -- using threads adds a little CPU overhead in exchange for reduced memory consumption. If you have high-memory-usage shared (e.g., global) variables you may see some memory usage benefits. But it is going to depend enormously on the design of your application. Even if other people running fastapi under high load could comment, I don't think they'd be able to give useful advice on these nuanced points without a relatively deep understanding of your application.

I suspect that for most typical fastapi applications, CPU will be more of a bottleneck than memory under high load (again, depends on your application). In that case, using only process-based workers should give the best performance.

[sandys]

@dmontagu well this seems to be one of those things that we are not getting answers anywhere.

For example, take a look another gunicorn configuration issue - #253 - where fastapi does not play well with per-core settings when run inside docker/kubernetes.

It is my humble opinion - that production configuration is a hugely dangerous thing here. And could become one of those things that hurt us after we have heavy investment in writing fastapi. Because our servers are being under-utilized (multi-core issue) or other problems.

We see that Uber, etc do use fastapi in production. It would be really awesome if we could get some learnings/rules-of-thumb from them.

[dmontagu]

The problem in the issue you linked had nothing to do with the specifics of fastapi or it ignoring settings related to pet-core settings. It was a timeout issue due to a large query, which would have happened with any server.

Again, I don’t think there’s anything specific to FastAPI here in terms of what a production config should look like.

[dmontagu]

Also, the other issues discussed there with workers restarting are not resolved. I don’t believe that is a production gunicorn config issue, but an issue of cloud config (which is generally going to be idiosyncratic to each application) and/or just bugs in people’s code.

[sandys]

@dmontagu i disagree here. there's no cloud config here, unless you are referring to docker config. however, given that we are seriously moving to production with fastapi and these are issues we are hitting...im not sure where else should i cry for help.

there is a similar issue being discussed of in context of threads - #603 (comment) . also directly related to production configuration.

[tiangolo]

@sandys as I said before, you shouldn't "enable" threads in Gunicorn apart from setting the number of Uvicorn workers.

Gunicorn was created for WSGI applications. It doesn't have a notion of ASGI. So it was made expecting that threads would be the only way to handle concurrency.

FastAPI uses an async event loop, that can be provided by Uvicorn (running uvloop underneath).

FastAPI takes care of running synchronous functions in a threadpool using run_in_executor.

So, FastAPI is already starting threads for you, for each running process (for each app instance).

Being able to handle concurrency directly and the full even loop with uvloop is what gives all the performance benefits. So, starting extra threads in Gunicorn (apart from the ones that each FastAPI instance is already starting) would at best take control out from uvloop, reducing performance, and at worst, could generate issues if you have context-unsafe code.

If you want to hyper-optimize performance more than that, you can check the internals of uvloop, read the docs for asyncio, the PEP for ContextVars and understand how it all works together and is integrated.

---

Also, let me ask you to double-check your messages. They are looking very critical and demanding. Try to make them more deferent. Everyone here is trying to help you. @euri10, @dmontagu and @devsetgo are just trying to help you, for free. We don't really earn anything from helping you. So, please, be respectful and gentle to the community, that's the least you can do.

[github-actions]

Assuming the original issue was solved, it will be automatically closed now. But feel free to add more comments or create new issues.

[SagiMedina]

Hey @sandys ,
After a year, any learning points? conclusion regarding using uvicorn and gunicorn in production?
recommendation?

[Gatsby-Lee]

@sandys I guess your expectation for FastAPI community is not quite higher.
At least, I appreciate to your effort to get what other ppl might look for.
Due to your effort, at least I can see more detailed explanation from @tiangolo :)

Thank you.

[Gatsby-Lee]

@tiangolo
Thank you very much for your detailed explanation.
It really helps :)