Implement keeping pooled connections open #974
Conversation
…on time is based on last use rather than created. Under heavy loads now, some connections may never expire.
This commit also fixes two preexisting issues: - the status of a connection wasn't checked in `getconn`, so that method could return a broken connection - the `putconn` method didn't reopen new connections if the total had dropped below `minconn`
|
@dvarrazzo Please let me know if you would prefer that I release this as a separate package. |
|
I've created a separate package: https://pypi.org/project/psycopg2-pool/. |
|
Hi, thank you for working on this issue, notwithstanding my lacklustre direct interest. Bit of a background: I hate psycopg connection pool. I don't think it does the right things. Its origin probably comes from Zope and its footprint is still there - and it's likely wrong outside of it. I think a good connection pool can be passive like psycopg one, if needed, but it might even do some smart, and have some form of internal keepalive and a better algorithm to gauge its size depending on its usage - in general be a bit bolder with its own behaviour. If done right it would probably break the psycopg pool interface, and I really see that module a bit of a legacy thing. I think it would be a good idea to start from scratch and have a pool module of its own and a pool interface independent from psycopg pool, as well as having a release cycle independent from psycopg. I'm glad you did the first step on your own. If you would like collaboration for that module's design (breaking from bloody Cheers -- Daniele |
|
@dvarrazzo Feel free to open an issue in https://github.com/Changaco/psycopg2-pool so that we can discuss your ideas. I didn't start from scratch, but I did remove the weird legacy stuff from the API and implementation. |
|
@dvarrazzo Nominating something derived from this PR for psycopg3 🙂 |
This PR is a rebased and refactored version of #565.
When a connection is added (via
_connect()) or returned (via_putconn()) to the pool, a timestamp is saved. When_getconn()pulls a connection, it compares the saved timestamp to the current time, and if the connection has been idle for longer thanidle_timeout, then that connection is closed and discarded.I've set the default value of
idle_timeoutto0in order to replicate the current behavior, however another value could be chosen if there is consensus on changing the default behavior.My first commit fixes two bugs in the current pool implementation:
_getconn()method doesn't check the status of a connection, so it can return a broken connection._putconn()method doesn't reopen new connections if the total has dropped belowminconn.