Allow configurable connection pool

[raymondjacobson]

Trello Card Link

Description

Moves connection pool to a config var, currently deployed on staging w/ 2000 max connections.
I'm somewhat convinced that we need to just dramatically increase our max connections. I've noticed that during high load, our RDS instance is totally fine but it seems like simple endpoints like /health_check struggle to grab a connection.
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.MaxConnections
We can definitely up this number a lot, though I'm not sure at which point node is just unable to utilize. I think it's worth a shot on prod though.

Services

• Creator Node

Does it touch a critical flow like Discovery indexing, Creator Node track upload, Creator Node gateway, or Creator Node file system?

Delete an option.

• ✅ Nope

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
Include log analysis if applicable.

1. Currently running on staging cn 3 @ 2000. Used locust to load test, saw no noticeable difference in performance when just spamming /health_check, but did see that more than 100 connections were used (500 concurrent requests ended up using 170 connections).

[dmanjunath]

I'm game to try setting a much higher number and seeing what happens

[jowlee]

I'm not sure if this is the issue b/c if you look at the RDS performance insights for creator nodes, it's showing a max of 50 concurrent active sessions. I think we need to increase the size of the rds instance w/ more CPU, we surpassed the vCPU which queues the requests, I believe

[raymondjacobson]

I'm not sure if this is the issue b/c if you look at the RDS performance insights for creator nodes, it's showing a max of 50 concurrent active sessions. I think we need to increase the size of the rds instance w/ more CPU, we surpassed the vCPU which queues the requests, I believe

It doesn't make sense to me that under heavy load requests to the db take a long time from the web server and not from navicat/psql. This to me screams "need more connections"

[jowlee]

Yeah, looks like we are maxing out the num connections too.

[raymondjacobson]

Yeah, looks like we are maxing out the num connections too.

Each of those dips is the server crashing