Research/document gunicorn/gevent settings

[lbeaufort]

It looks like our applications can start to suffer performance issues when the database is only around 30% CPU. This could be a concurrency issue - I don't know whether this is the best approach for addressing the issue, but we should still research/document gunicorn/gevent settings. I don't know for sure how many concurrent requests we can handle for each app instance.

Helpful article: https://medium.com/building-the-system/gunicorn-3-means-of-concurrency-efbb547674b7
Docs:

• https://docs.gunicorn.org/en/19.10.0/design.html
• http://www.gevent.org/contents.html
• https://stackoverflow.com/a/41696500

To use Async [gevent], you need to be sure that your processing is not compute bound, this means you will not be able to make use of multiple cores. Advantage you get is that the memory that multiple threads would take would not be there. But you have other issues like non monkey patched libraries. Move to Async only if the threaded worker does not meet your requirements.

With gevent:

I would be scared of what modules might not be monkey patched correctly in order to work predictably when Async worker class is used. Async for all its benefits does come with its own risks. You must make absolutely sure that all your code is monkey patched and no native code runs.

• Example of gevent blocking: https://tech.wayfair.com/2018/07/blocking-io-in-gunicorn-gevent-workers/

Gevent is an alternative to threading where, instead of spawning threads, it runs a libev event loop and monkey patches Python to replace most blocking IO with non-blocking IO. This allows you to write Python in a synchronous style without the complexity of managing threads, while also reaping the benefits of non-blocking IO.

Gevent achieves this concurrency using greenlets, which are a “lightweight pseudo-thread” that work cooperatively via an event loop to yield control to one another while they are waiting for IO. Looking into Gunicorn’s GeventWorker, you can see where it runs Gevent’s monkey patch on startup to replace blocking IO with non-blocking IO.

Gevent and Gunicorn try their best to monkey patch blocking IO in the Python standard library, but they can’t control external C dependencies. This becomes a serious issue in web apps; if your event loop is blocked waiting for a C libraries’ IO, you can’t respond to any requests, even though you have plenty of system resources available.

beware of blocking IO in Gevent applications! It causes non-intuitive and difficult-to-diagnose performance issues.

• https://www.projectcalico.org/the-sharp-edges-of-gevent/
• Why You Should (Almost) Always Choose Sync (not gevent) Gunicorn Workers

Completion criteria

• Consider reaching out to cloud.gov for support hours
• Read the docs for version we're using and most recent stable version
  

  openFEC/requirements.txt

  Lines 19 to 20 in ce36d69

  	gunicorn==19.10.0
  	gevent==1.4.0

• Determine our concurrency capabilities
  • Identify how to measure and log - Kibana report of requests per second?
• Consider adding gunicorn threads
• Open up follow up tickets to test any changes using locust

[lbeaufort]

I checked the PI planning document and this is slated for 13.2

[patphongs]

Moving to 15.6 due to resource constraints

[patphongs]

Research has been done with this, however, measuring per second requests has been a blocker. We'll revisit when we have more resources in the feature.