sqlalchemy.exc.TimeoutError: QueuePool limit of size 8 overflow 8 reached, connection timed out

[sarvajit22]

We are facing this issue when large dashboards with multiple charts are refreshed. We tried tuning pool size but that did not help much,

sqlalchemy.exc.TimeoutError: QueuePool limit of size 8 overflow 8 reached, connection timed out, timeout 90.00 (Background on this error at: https://sqlalche.me/e/1>

current config:

SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 8,
    "max_overflow": 8,
    "pool_pre_ping": True,
    "pool_recycle": 1800,
    "pool_timeout": 90,
}

Gunicorn | gevent, 10 workers ×250 connections
Celery | 2×6
Postgres | max_connections = 200

Postgres & redis same host as superset

[darekwojciechowski]

That error is about Superset's metadata DB pool, and it's a per-worker pool. Each gunicorn worker has its own pool of pool_size + max_overflow = 8 + 8 = 16 connections. When a big dashboard refreshes, one worker fires many chart queries at once (each needs a metadata connection for logging/results), blows past 16 concurrent, and the 17th waits pool_timeout=90s then throws.

Raising the pool "didn't help much" because you also have a hard ceiling: total connections = workers × (pool_size + max_overflow). With gevent and 10 workers you must keep that under Postgres max_connections=200, minus Celery's own usage. 10 × 16 = 160 today; you can't just crank pool_size to 50 (10 × 100 = 1000 >> 200).

Two ways forward:

1. Right-size within the ceiling. Fewer, fatter workers give each one a bigger pool without exceeding Postgres. For example 5 workers:
   SQLALCHEMY_ENGINE_OPTIONS = {
   "pool_size": 15,
   "max_overflow": 15,
   "pool_pre_ping": True,
   "pool_recycle": 1800,
   "pool_timeout": 30,
   }
   5 × 30 = 150 connections, but each worker now tolerates 30 concurrent chart queries instead of 16. Also bump Postgres max_connections (e.g. 300) if you have RAM.

2. The scalable fix: put PgBouncer in front of the metadata Postgres in transaction pooling mode. Then hundreds of gevent greenlets share a small set of real Postgres connections, and pool exhaustion basically goes away. This is the standard answer for gevent workers + Postgres connection explosion.

For a quick unblock use option 1; for a real load pattern, do option 2.