Airflow 3.2.0 opens too many meta DB connections

[rbayer666]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.0

What happened and how to reproduce it?

** Issue Description **
After upgrading from Airflow 3.1.8 to 3.2.0 without changing the configuration we encountered the problem that the meta database (running on PostgreSQL) stops accepting connections. Or, if we increase the allowed connections on the database, the sql achemy pool fills up.

The sql_alchemy settings are the default ones:

• sql_alchemy_pool_enabled = True
• sql_alchemy_pool_size = 5
• sql_alchemy_max_overflow = 10

The allowed parallel database connections on the postgresql is set to 100.

We have 1x api server, 1 x scheduler, 1 x triggerer.

Increasing the sql_alchemy pool and the postgresql connections does not solve the issue - it just takes more time until the connection pool is filled up or the database stops accepting connections.

It looks like the database connections are not properly closed and left open. Over time the connections sum up.

What you think should happen instead?

All database connections should be closed when they are no longer needed.

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Docker-Compose

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

v1.31.13

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[alessandro-peyrachia-rnc]

Hi, I noticed something similar, we just upgraded a couple hours ago.
Even with pgbouncer in front of the postgres db, api will fail after a certain time.
We have 2 api and 2 scheduler, 1 triggerer, 3 workers.

core:
        parallelism: 128
        max_active_tasks_per_dag: 64
      scheduler:
        max_tis_per_query: 256
      database:
        sql_alchemy_pool_size: 5
        sql_alchemy_max_overflow: 10
        sql_alchemy_pool_recycle: 600

pgbouncer set at

poolMode: transaction
parameters:
      max_client_conn: "1000"
      default_pool_size: "30"
      min_pool_size: "10"
      reserve_pool_size: "10"

Affects mainly the api service

[dclfan]

Hi, same experience with the upgrade from 3.1.8 to 3.2.0. We tried on Monday but our test suite was able to replicate this problem in about two hours. We reset everything and tried again, and at around ~2hrs we had DB connection issues that failed all of the jobs in flight. It looked like it recovered on its own if given enough time ~10 min but we had to rollback to 3.1.8. We use pgbouncer, as well.

pgbouncer settings

pool_mode = transaction max_client_conn = 1000 default_pool_size = 20 max_db_connections = 75

[kaxil]

To help narrow down the root cause, could you capture some PostgreSQL diagnostics next time the issue occurs?

1. Connection state snapshot (most important -- run when connections are high):

SELECT state, count(*) AS cnt
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state
ORDER BY cnt DESC;

And the detailed view:

SELECT state, wait_event_type, wait_event,
       left(query, 100) AS query_prefix,
       age(now(), query_start) AS duration
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY query_start;

This tells us whether the stuck connections are active, idle, or idle in transaction.

2. Lock contention -- are connections waiting on each other?

SELECT blocked.pid, left(blocked.query, 80) AS blocked_query,
       blocking.pid AS blocking_pid, left(blocking.query, 80) AS blocking_query
FROM pg_stat_activity blocked
JOIN pg_locks bl ON bl.pid = blocked.pid AND NOT bl.granted
JOIN pg_locks gl ON gl.locktype = bl.locktype AND gl.relation = bl.relation AND gl.granted
JOIN pg_stat_activity blocking ON blocking.pid = gl.pid
WHERE blocked.datname = current_database();

3. Log table size -- the log table gets an INSERT on every task state change (new in 3.2.0). If it's large, INSERTs slow down and hold connections longer:

SELECT relname, n_live_tup, n_dead_tup,
       pg_size_pretty(pg_total_relation_size(oid)) AS total_size,
       last_autovacuum
FROM pg_stat_user_tables
WHERE relname = 'log';

4. Connection growth trend -- run every 5 minutes for ~30 minutes to capture the accumulation pattern:

SELECT now(), count(*), state
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state;

5. PostgreSQL timeout settings:

SHOW idle_in_transaction_session_timeout;
SHOW tcp_keepalives_idle;

The state column from is the single most useful data point. If we see growing idle in transaction connections, that's a session leak. If we see active connections piling up on INSERT INTO log or SELECT ... FOR UPDATE, that's lock contention from new 3.2.0 query patterns.

[dedalozzo]

I noticed the same. When I run it locally, on Docker, I use this configuration:

AIRFLOW__CORE__PARALLELISM: 4
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 2
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 2
AIRFLOW__API__WORKERS: 1
AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES: 1
AIRFLOW__CELERY__WORKER_CONCURRENCY: 1

With the same configuration, Airflow 3.2.0 is completely unusable. Same issue on our dev environment with standard configuration and 8 GB of RAM. Immediately exhausted all the connection from the pool.

Airflow 3.2.0 has critical performance regressions: 14-second DAG parse times, Task SDK connection pool exhaustion, triggerer burning 100% CPU with 0 triggers, and worker memory bloat (2.5 GB idle)!

Task SDK rewrote the execution model, introducing HTTP API calls for every task lifecycle event. This saturates DB connection pools and causes cascading timeouts across all components. No tuning combination resolved the issue.

It is an architectural disgrace. Why did you release something like that? Release 3.1.9 at least, fix the JWT vulnerability. And then you can take 6 months to rework 3.2.0.

[potiuk]

It is an architectural disgrace. Why did you release something like that? Release 3.1.9 at least, fix the JWT vulnerability. And then you can take 6 months to rework 3.2.0.

Not sure if you are aware, but It's been at least 4 weeks testing beta2 and rcs - and @dedalozzo - I have not seen you testing it ? Or did I miss it? It's a bit disgrace to not see you testing it and complaining after. Did I miss your issues and testing reports? Have you send them our way? Maybe they were lost somewhere in spam?

Stating some issues with the new release would have been helpful before we released, apparently however you missed your opportunity to give back for the free software you get from people who created it.

Unfortunately, it's likely something that affects your configuratiion - and I believe It would really help if you follow what @kaxil asked for rather than throwing invectives at people.

What might also help is increasing the resources while testing and seeing if it helps - and then reports the details asked, in case you have problems with your 8GB. Or just wait until others will. I assume you are talking about CVE-2025-57735 - JWT issue was a low one, we usually would not announce a critical issue without waiting to see if we need to upgrade to a new versio. However, I am wondering (because releasing older versions of airflow with backported fixes, costs an extra time) - we are discussing how we can respond to the expectations of our users, to have older versios with security fixes, and I wonder - maybe you or your company would like to pay extra for it? We are currently discussing about changing vulnerability handling for open-source and paying for some kind of priority security for older versions seems like an option we consider - because obviously you realise that you ask for more money for the volunteers to backport a low-severity issue to older version - you do realise that this is what you ask for ?

Yes we might have some teething issues in 3.2.0, but it's a bit of a bold statement stating disgrace for people who invest their soul and volunteer time to implement something, you can take for free, without any guarantees or expectations - onlyot by seeing few cases where connection usage grew. It's a bit of overstatement.

Yes, It might turn out that some more resources are needed in 3.2.0 We do not guarantee you will still be able to use exactly the same resources, and maybe there is a butg or two that will be fixed as soon as people like you who decide to help rather than throw invectives at open-source developers who contribute often their free time, so that you can use the software for free. We are all humans, we make mistakes - no-one is free of them - and I think when you get something for free, helping to solve your problem so that you can get fixed versions for free is probably. best course of action.

I understand that you might have bigger expectations here, but if you consider how much you paid (0) for the software, your expectations should be somewhat adjusted - especially that you have not helped to test the release when we asked.

However, it's not too late., You can still help. I think there are quite few constructive ways you can do here:

a) (a bit too late - you missed it) - test a new version when beta/rc releases come out (but you can catch up next time)
b) provide helpful diagnostics so that people who spend their free time to be able to find the issues and release fixed version quickly/

When we will ask for testing a new version (which usually last 3 days and have an issue where you can install and test things) - you can even verify if the issue is fixed when it will be. Highly recommended as it will allow us to release better software and avold more disgruntled and complaining users.

Genereally - I personally think - being nicer to people who spend their weekends and nights away of their family and friends so that they can produce a software you can use for absolutely free, is a very good idea. It usually works better for example when you need help in the future - which (providing that you are nicely asking for help and show you've done your part) you might sometimes get from them. But also when you have a problem, those people might be exactly those who might help to solve your problem @dedalozzo. This is usually what most people do - only few have unrealistic expectations, and demands "paid" level of services from open-source and free projects, and I guess being part of the community and majority is a good idea.

I think it would be great if you try to actually help in solving the issue - following the ask from @kaxil, or at the very least encourage others to provide enough info and express your gratitued for the software you have and at the very least cheer-lead those who work on making your issues disappear.

That woudl be my suggestion @dedalozzo

[amarbrkic]

Hi, I had similar issues when upgrading to versions 3.1.2 and 3.1.3. I can't guarantee this will solve your issue(s). I solved mine by doing the following:

1. Fixing my SSL issue. I would recommend checking the logging in your webserver log file, that's where I found my issues.
2. Changing the file parsing sort mode if you are using mnt on your containers. It seems like parsing multiplies connections, but unfortunately I haven't found out why. I have changed my file_parsing_sort_mode to alphabetical.

[LengYue12389]

Hi, I seem to have encountered the problem of running out of connection pool, and I haven't found the cause yet

[alessandro-peyrachia-rnc]

Hi @kaxil ,
I have gathered the output required, have a look here below.
I'm available to dig deeper.

app=> SELECT state, count(*) AS cnt
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state
ORDER BY cnt DESC;
 state  | cnt
--------+-----
 idle   |  20
 active |   1
(2 rows)

app=> SELECT state, wait_event_type, wait_event,
       left(query, 100) AS query_prefix,
       age(now(), query_start) AS duration
FROM pg_stat_activity
WHERE datname = current_database()
ORDER BY query_start;
        state        | wait_event_type | wait_event |                                             query_prefix                                             |     duration

---------------------+-----------------+------------+------------------------------------------------------------------------------------------------------+--------------
----
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:04:23.4034
78
 idle                | Client          | ClientRead | ROLLBACK                                                                                             | 00:04:18.3534
14
 idle                | Client          | ClientRead | ROLLBACK                                                                                             | 00:04:17.9472
52
 idle                | Client          | ClientRead | SELECT 1                                                                                             | 00:04:17.6435
47
 idle                | Client          | ClientRead | ROLLBACK                                                                                             | 00:03:48.3836
62
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:03:47.8090
68
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:03:47.7953
92
 idle                | Client          | ClientRead | SELECT ceil(COUNT(*)::decimal / 32) FROM task_instance WHERE (state='running' OR state='queued') AND | 00:03:47.7107
23
 idle                | Client          | ClientRead | ROLLBACK                                                                                             | 00:03:47.6354
18
 idle                | Client          | ClientRead | ROLLBACK                                                                                             | 00:03:47.6234
39
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:03:43.7063
49
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:03:43.7055
76
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:03:21.2596
13
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:00:31.7996
02
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:00:02.4373
78
 idle                | Client          | ClientRead | SELECT ceil(COUNT(*)::decimal / 32) FROM task_instance WHERE (state='running' OR state='queued') AND | 00:00:01.9965
34
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:00:00.3019
44
 idle                | Client          | ClientRead | COMMIT                                                                                               | 00:00:00.0795
6
 active              |                 |            | SELECT state, wait_event_type, wait_event,                                                          +| 00:00:00
                     |                 |            |        left(query, 100) AS query_prefix,                                                            +|
                     |                 |            |        age(now()                                                                                     |
 idle in transaction | Client          | ClientRead | DELETE FROM dag_warning USING dag WHERE dag_warning.dag_id = dag.dag_id AND dag.is_stale = true      | -00:00:00.001
897
 idle                | Client          | ClientRead |                                                                                                      |
(21 rows)

app=> SELECT blocked.pid, left(blocked.query, 80) AS blocked_query,
       blocking.pid AS blocking_pid, left(blocking.query, 80) AS blocking_query
FROM pg_stat_activity blocked
JOIN pg_locks bl ON bl.pid = blocked.pid AND NOT bl.granted
JOIN pg_locks gl ON gl.locktype = bl.locktype AND gl.relation = bl.relation AND gl.granted
JOIN pg_stat_activity blocking ON blocking.pid = gl.pid
WHERE blocked.datname = current_database();
 pid | blocked_query | blocking_pid | blocking_query
-----+---------------+--------------+----------------
(0 rows)

We use cnpg with posgres 18, oids are not enabled by default

app=> SELECT relname, n_live_tup, n_dead_tup,
       pg_size_pretty(pg_total_relation_size(oid)) AS total_size,
       last_autovacuum
FROM pg_stat_user_tables
WHERE relname = 'log';
ERROR:  column "oid" does not exist
LINE 2:        pg_size_pretty(pg_total_relation_size(oid)) AS total_...

app=> SELECT now(), count(*), state
FROM pg_stat_activity
WHERE datname = current_database()
GROUP BY state;
              now              | count | state
-------------------------------+-------+--------
 2026-04-13 08:48:38.289699+00 |     1 | active
 2026-04-13 08:48:38.289699+00 |    20 | idle
(2 rows)

app=> SHOW idle_in_transaction_session_timeout;
SHOW tcp_keepalives_idle;
 idle_in_transaction_session_timeout
-------------------------------------
 0
(1 row)

 tcp_keepalives_idle
---------------------
 7200
(1 row)

I run the queries directly on the db, not through pgbouncer

What is unusual is the behaviour when job gets queued.
This morning we added a batch of around 200 jobs, using a pool of 20 slots
The db showed this behaviour here, which was not present before

Let me know how can I help further,
Thanks