Performance testing and fine tune: API and CMS

[lbeaufort]

@patphongs commented on Thu Apr 30 2020

What we're after: In order to ensure the best timeout settings in the CMS, we need to perform load testing in the API and check how long it takes for the CMS to respond.

Completion criteria:

• Determine an optimal timeout for CMS api_caller. Make sure we know/document retry settings to better understand expected behavior.
• Set up API and CMS app instances to match production
• Test performance with 2 and 4 Aurora db clusters We set up autoscaling
• Document locust results - share a link to the results here, or how to get to them
• Open up follow up issues for additional changes needed, including one to tie to the IR documentation

Technical steps

Setup

• set up the environment: will need to work with database team to replicate prod DB setup. set up application instances for API and CMS

Run the tests

• Replicate 4/15 load in API and CMS
• Test the API alone, document results
• Test the CMS and API together, document results

Cleanup

• Put everything back (DB, application instances

[lbeaufort]

Followup tickets, tagged for 12.6:

• Add locust to CMS fec-cms#3792 Add locust to CMS
• Research: Too many connections error with high CMS load fec-cms#3791 Too many connections error with high CMS load
• Research/document gunicorn/gevent settings #4382 Research/document gunicorn/gevent settings