AWS RDS Proxy (Postgresql) is pinning requests for undetermined reasons #13148
|
Hello, We are using RDS Proxy to manage connection pooling and load balancing for db server instances. One benefit of using rds proxy is that it will split transactions to different db connections. I think in practice it means that if our application creates 10 connections, but not all are actively writing, we would need < 10 connections between the RDS proxy and RDS Postgres. However we are seeing that RDS proxy is pinning connections from our application to specific RDS proxy connections. They have a list of guidelines to avoid in order to prevent having "stateful" connections: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-proxy-pinning.html#rds-proxy-pinning.postgres We are not using any of the queries that they list in that article. I have turned on sqlalchemy query logging, there are no SET statements, or any of the other ones listed in that article. I am using psycopg2, and I have also tried configuring the connection to use the NullPool, and it still is presenting the same issue.
Is there anything missing from logging that would make it difficult to see session state changes? |
Replies: 3 comments 13 replies
|
One last thing to note, is that the query it is complaining about, is after a commit, and the ORM model for user is refreshed in a new transaction. |
I assume this means, if you run three transactions on a python-side connection, it may use three connections on the server side. This is what PGBouncer also does, though this mode causes a lot of problems with prepared statements. note that all the PG drivers other than psycopg2 use prepared statements, and that psycopg2 itself is legacy (it's been superseded by psycopg, which uses prepared statements). so if this proxy layer cannot handle PREPARE at all, you are not in a good situation for the future.
right so you should have very little going on that would get in the way of such proxies. there is one SET we have and we use it to set the transaction isolation level, but only if you are specifying a transaction isolation level other than the default (this statement is not logged client side, however). we also use psycopg2 itself might be emitting other kinds of SET so you might want to try experimenting with a plain psycopg2 connection. |
|
Just to follow up here. The final response from AWS was: If someone else encounters this before aws fixes the bug. We mitigated it by setting some_reset = Column(..., quote=True) |
Just to follow up here. The final response from AWS was: