Research: Too many connections error with high CMS load

[lbeaufort]

Found this through locust testing - it's hard to know exactly how locust will map to real-time usage. With 4 application instances, even with a 30 second timeout in the API caller, I was still getting this error with 1500 users in Locust (about 130 requests/second when I've seen peaks around 230/seconds), which results in 500 errors.

Action items:

• We should research: connection pooling, closing connections, adjusting timeout, and reducing the number of CMS application instances, gunicorn, etc
• We can test this with locust. (Tested with this closed PR (don't merge): [WIP] Add 4/15 queries #3784 but we'll need to put in a legitimate PR to add locust)

Completion criteria:

• Based on research trying to reach threshold of 230 requests/second, create followup tickets to address the error

  2020-05-27T17:30:34.71-0400 [APP/PROC/WEB/3] ERR ERROR:django.request: Internal Server Error: /data/elections/house/NJ/03/2020/
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR Traceback (most recent call last):
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.connect()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 195, in connect
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.connection = self.get_new_connection(conn_params)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     connection = Database.connect(**conn_params)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR psycopg2.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR The above exception was the direct cause of the following exception:
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR Traceback (most recent call last):
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     response = get_response(request)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/utils/deprecation.py", line 93, in __call__
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     response = self.process_request(request)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/wagtail/core/middleware.py", line 13, in process_request
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     request.site = Site.find_for_request(request)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/wagtail/core/models.py", line 116, in find_for_request
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     return get_site_for_hostname(hostname, port)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/wagtail/core/sites.py", line 35, in get_site_for_hostname
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     'root_page'
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/models/query.py", line 274, in __iter__
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self._fetch_all()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self._result_cache = list(self._iterable_class(self))
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1138, in execute_sql
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     cursor = self.connection.cursor()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 256, in cursor
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     return self._cursor()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 233, in _cursor
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.ensure_connection()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.connect()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     raise dj_exc_value.with_traceback(traceback) from exc_value
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 217, in ensure_connection
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.connect()
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 195, in connect
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     self.connection = self.get_new_connection(conn_params)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 178, in get_new_connection
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     connection = Database.connect(**conn_params)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR   File "/home/vcap/deps/0/python/lib/python3.7/site-packages/psycopg2/__init__.py", line 127, in connect
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   2020-05-27T17:30:34.55-0400 [APP/PROC/WEB/1] ERR django.db.utils.OperationalError: FATAL:  remaining connection slots are reserved for non-replication superuser connections

[patphongs]

Logging research associated with this ticket here: https://github.com/fecgov/fec-accounts/issues/275

[patphongs]

Research is complete, closing in favor of this follow-up testing issue: #3872