fides >= 2.18.0 maintains a large number (~50) of open idle connections to hosted database #3918
Comments
|
This might be related to @ThomasLaPiana, the date for when you merged #3884 roughly lines up with db conn count increase I'm seeing. Do you think that the changes you made could cause a lot of idle open db conns? |
|
@daveqnet absolutely, it should be a one-line fix! |
|
@daveqnet I'm fixing this on the application side hopefully, but can we also configure postgres to recycle connections more often too? Defense on both sides seems prudent |
|
@daveqnet I'm having a hard time reproducing this locally. When I run off of current code I see ~5 connections maintained. The I am seeing however that the application has a limit of 50 with an additional 50 set as overflow. Maybe something is keeping application connections open? Is anything happening on that instance other than health checks? |
@ThomasLaPiana From an architecture perspective we'll probably have to put something like AWS RDS Proxy in front of shared db instances for multiple tenants, but that'll take a while to deliver. In the meantime, yes, there's absolutely some config changes that can be made on the shared postgres db side. For example, the idle connection timeout is currently set to 24 hours (86400000 ms). On local fides deployments this is set to 0 (no timeout limit). Get timeout SELECT name, setting, unit
FROM pg_settings
WHERE name = 'idle_in_transaction_session_timeout';I can set this timeout to, say, 30 minutes (1800000 ms). Based on the db conn durations I'm seeing, this would reduce the db conn count by 90%. |
@ThomasLaPiana Honestly the only requests I'm seeing in the fides webserver logs are automated check requests to However, nightly is a little unusual in that it has two workers. Do workers connect to the host db, or just the webserver? |
|
Just to add some additional info:
worker error log |
daveqnet
changed the title
daveqnet
changed the title
daveqnet
changed the title
I was wrong on this. This timeout only ends connections in the @ThomasLaPiana, there's no config we can set on the fides containers to limit their connections client-side, is there? |
|
@daveqnet our docs say there is: https://ethyca.github.io/fides/2.18.0/config/ (ctrl + f for We default to |
|
Aha, yes, I should probably check the config docs before asking a config question!! 🤦 Thanks @ThomasLaPiana 🙏 |
|
I worked hard to write a script to auto-generate that documentation 🥲 |
I'm genuinely going to add it to my bookmarks toolbar 😆 |
😂 ❤️ 💯 |
Bug Description
Since August 3rd the nightly build (and to a lesser degree the rc build) of fides on our staging environment is keeping a large number of idle database connections open. It's about a 3x increase.
[edit: I've reproduced this on fides 2.18.0]
Occasionally this can overwhelm our database instance, making it unavailable for other staging tenants.
See additional context section below for more detail.
Confidential & internal data about the infra that I don't wish to include on a public issue can be viewed in the associated OPS ticket.
Steps to Reproduce
Expected behavior
The number of connections would be much lower, e.g. 1/3 of observed.
Screenshots
n/a
Environment
Additional context
Query: how many connections are open?
Query: how many connections are active?
Query: how long have the connections been open?
Query: where are the connections being made from?
Note: I've redacted the IP address, but it's a single address.
Query: What are non-idle connections being used for?
The text was updated successfully, but these errors were encountered: