Production configuration of gunicorn

[sandys]

NOTE: I'm not sure whether i should be filing this here or upstream on starlette. Please let me know if Starlette developers are more knowledgeable about this. However, I'm using fastapi.

What is a good configuration setting for gunicorn in production.

On reading documentation of gunicorn with uvicorn, there are lines like this

"Gunicorn provides a different set of configuration options to Uvicorn, so some options such as --limit-concurrency are not yet supported when running with Gunicorn"

Is it possible for you to share a list of options that work well ? One of the biggest question is around threads - should we set the threads option?
what about stuff like "worker_connections" , "keepalive" (especially in context of async code)

[dmontagu]

This might be a reasonable starting point: https://github.com/tiangolo/uvicorn-gunicorn-docker/blob/master/python3.7/gunicorn_conf.py

But it doesn't go into much depth on the available options.

The starlette and uvicorn repos would also be good places to ask this question / search for related issues.

[devsetgo]

I started mine off from a gunicorn example https://github.com/benoitc/gunicorn/blob/master/examples/example_config.py

Then set mine like this - https://github.com/devsetgo/test-api/blob/master/app/gunicorn_cfg.py

From my experience, i have found that I really just need to set workers, bind, port, and for starlette/fastapi the worker class is what makes uvicorn work with gunicorn.

[sandys]

Hi mike Are you setting threads in gunicorn ? Im not sure if the Uvicorn worker plays well with threads. What's the things you tune for in production?

…

On Sun, 22 Sep, 2019, 05:29 Mike Ryan, ***@***.***> wrote: I started mine off from a gunicorn example https://github.com/benoitc/gunicorn/blob/master/examples/example_config.py Then set mine like this - https://github.com/devsetgo/test-api/blob/master/app/gunicorn_cfg.py From my experience, i have found that I really just need to set workers, bind, port, and for starlette/fastapi the worker class is what makes uvicorn work with gunicorn. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#551>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU67APO7ZOC4ELT2LWDQK2YONANCNFSM4IYM5USQ> .

[devsetgo]

The item I am most concerned with for my production use are the number of workers. For async I've tried a couple different worker classes, but didn't see much change. So I just use uvicorn.workers.UvicornWorker. I set the backlog of connections, but I haven't ever found it to be an issue.

In my opinion, I don't think threads will matter. Threads I presume will matter more if you are kicking on new threads using multi-threading. I've not tried threads with Gunicorn for sync or async, so that could be an incorrect assessment.

I've done some testing of my test-api using WRK and connecting to my server 1000 miles away (1600 Km). So far with 9 workers running (4 cores x 2 + 1) I max out just below 6k requests per second. I find that Traefik and Let'sEncrypt drops that to 2k (research area). A single core (1 x 2 + 1) I can get around 2k requests per second (drops to 1k with Traefik/LetsEncrypt). I eventually plan to do more tests, but this is good enough for my use and horizontal scaling would increase this easily.

[sandys]

Thanks for that info. What tuning are you doing for backlog? Would love to know learnings here. P.S. we are using haproxy - we find the config to be simpler and much higher performance.

…

On Sun, 22 Sep, 2019, 23:04 Mike Ryan, ***@***.***> wrote: The item I am most concerned with for *my* production use are the number of workers. For async I've tried a couple different worker classes, but didn't see much change. So I just use *uvicorn.workers.UvicornWorker* . I set the backlog of connections, but I haven't ever found it to be an issue. In my opinion, I don't think threads will matter. Threads I presume will matter more if you are kicking on new threads using multi-threading. I've not tried threads with Gunicorn for sync or async, so that could be an incorrect assessment. I've done some testing of my test-api using WRK and connecting to my server 1000 miles away (1600 Km). So far with 9 workers running (4 cores x 2 + 1) I max out just below 6k requests per second. I find that Traefik and Let'sEncrypt drops that to 2k (research area). A single core (1 x 2 + 1) I can get around 2k requests per second (drops to 1k with Traefik/LetsEncrypt). I eventually plan to do more tests, but this is good enough for my use and horizontal scaling would increase this easily. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#551>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYUZLA7OCYKPTOCRBJBTQK6UA3ANCNFSM4IYM5USQ> .

[devsetgo]

Backlog setting came as a recommendation to add from a friend to add. The maximum is 2048 if I remember right, which is what I use as a setting. I've tried a few changes to the setting, but for my own use it has not made any difference.

Haven't used HAProxy, but I was using Nginx for the last 10 years and jwilder/nginx-proxy. Performance was pretty good (better than Traefik), but I find configuration is easier with Traefik + LetsEncrypt and Docker integration. So that is why I have been using it for the last 6 months.

Thanks for the tip on HAProxy, I'll look into it.

Edit - Love the name of your domain. Made me laugh... excellent choice.

[euri10]

I think most of your questions are answered quite well here @sandys https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7

[sandys]

Hi Thanks for replying . We already use gunicorn in a high load site with flask. We are very familiar with this. We also use uwsgi so probably have an entire gamut of understanding there. We do not understand gunicorn with Uvicorn workers. From what I understand, threads are recommended against. And the way toncreate pools+sessions is also very different. So the link you pointed out is not very helpful in this specific context.

…

On Wed, 25 Sep, 2019, 14:51 euri10, ***@***.***> wrote: I think most of your questions are answered quite well here @sandys <https://github.com/sandys> https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU2R625IGMR55V2TVWTQLMURHANCNFSM4IYM5USQ> .

[euri10]

So the link you pointed out is not very helpful in this specific context.

so sorry it didn't help @sandys , next time I'll google something more useful that will fit your exact needs, hopefully 😜

[sandys]

Oh I'm very thankful that you replied. It shows the help of the community and I thank you for that. Fastapi is very new and we are also trying to figure out production stuff

…

On Wed, 25 Sep, 2019, 18:16 euri10, ***@***.***> wrote: So the link you pointed out is not very helpful in this specific context. so sorry it didn't help @sandys <https://github.com/sandys> , next time I'll google something more useful that will fit your exact needs, hopefully 😜 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU4R2M6OQ5VYRLNSA7LQLNMSTANCNFSM4IYM5USQ> .

[dmontagu]

@sandys Thanks for your feedback/contributions! Please keep sharing as you figure out more of the production tuning stuff, I'm definitely hoping to gain some knowledge from these tuning efforts. (Sorry I can't be more helpful on this point...)

Have you tried asking in the uvicorn / starlette repos? I think there is a broader (and often very technically sophisticated) audience watching those repos.

[sandys]

yes i have asked - Kludex/starlette#642 but i havent gotten any replies yet.

…

On Wed, Sep 25, 2019 at 6:31 PM dmontagu ***@***.***> wrote: @sandys <https://github.com/sandys> Thanks for your feedback/contributions! Please keep sharing as you figure out more of the production tuning stuff, I'm definitely hoping to gain some knowledge from these tuning efforts. (Sorry I can't be more helpful on this point...) Have you tried asking in the uvicorn / starlette repos? I think there is a broader (and often more technically sophisticated) audience watching those repos. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#551>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU5EM4IPCEHRB4ZH3XDQLNOJVANCNFSM4IYM5USQ> .

[krnr]

I've been trying to enable pre_request/post_request hooks for Uvicorn worker, but with no avail. They are supported, but why their behavior is different from standard worker?

pre_request set up as usual

loglevel = "debug"
worker_class = "uvicorn.workers.UvicornWorker"

def pre_request(worker, req):
    worker.log.warning('foo')

has no effect. but set as

def pre_request(worker):
    worker.log.warning('UUUU')

results in error: Invalid value for pre_request: <function pre_request at 0x7feb8dee0a60>. so, it's processed, but how? looking at the debug output:

  pre_request: <function pre_request at 0x7f9bb01f5a60>
  post_request: <function validate_post_request.<locals>.<lambda> at 0x7f9bb01f5b70>

why post_request is a lambda function and how to set it?

when I use standard sync worker everything is fine.