Service Interruption: suspected OOM

[GalvinGao]

(investigate today)

[GalvinGao]

Seems that when there's too many (we are saying actually just 10-ish) connections to the Postgres DB, due to the fact that database/sql.DB is pooled by default, the connections are persisted and reused by the pool and each connection could elapse for an indefinite of time.

This created an issue on Postgres but the actual root cause is still unknown, as it sounds just not right to persist transaction data in the whole connection lifetime rather than just throw it away after the transaction completed. More investigation required.

A workaround (ada2b92) has been implemented as it limits the connection pool size on the backend side to reduce the connection amount been put on Postgres, but the root cause is still remain to be solved.

PS: A server-sided connection pool might be used but due to the pool itself is still elapsing a long time I doubt this would be the solution.

[FlandiaYingman]

What actually happened on the server? Was the service killed by OOM killer or panicked?

However, it's quite interesting and puzzling...

[FlandiaYingman]

My opinion would be to check the pg_stat_activity first. If there are more idle than active connections, pgBouncer or similar tools can be a big benefit.

[GalvinGao]

Fixed with SetConnMaxLifetime to a definite time instead of the unlimited default.