Scheduler stalls: 3.2 exceeds_max_non_backfill write contends with DAG-processor SELECT … FOR UPDATE on the dag table

[dor-bernstein]

Apache Airflow version

3.2.0+ (observed on 3.2.2)

What happened

The scheduler intermittently stalls: scheduler.critical_section_duration climbs while scheduler CPU drops and task scheduling starves. Inspecting the metadata DB during a stall shows scheduler statements of the form UPDATE dag SET exceeds_max_non_backfill = … WHERE dag.dag_id = … blocked with wait_event_type = Lock / wait_event = transactionid, waiting on the DAG processor's parse transaction.

Root cause

Before 3.2, the scheduler evaluated max_active_runs in memory and did not write the dag row during dagrun scheduling. #60006 ("Separate 'next dag run' from 'max active runs'", merge commit 32bc0119b408c9a37841edd021b646324a02745e, shipped in 3.2.0) added the dag.exceeds_max_non_backfill column (migration 0097_3_2_0_add_exceeds_max_runs_flag_to_dag_model) and made the scheduler write it to the dag row during dagrun scheduling:

• Write: airflow-core/src/airflow/jobs/scheduler_job_runner.py::_set_exceeds_max_active_runs (called from _create_dag_runs and _start_queued_dagruns).

That write now competes with a pre-existing lock held by the DAG processor for the entire per-file parse transaction:

• Lock: airflow-core/src/airflow/dag_processing/collection.py::DagModelOperation.find_orm_dags → with_row_locks(select(DagModel) …) i.e. SELECT … FOR UPDATE on every dag row for a parsed file, held through update_dags + serialization + permission sync until the transaction commits.

So the scheduler's UPDATE dag SET exceeds_max_non_backfill = … blocks on the DAG processor's FOR UPDATE. This is amplified when a single parsed file defines many DAGs (e.g. dynamically generated DAGs), because that parse transaction can run for minutes — holding the dag-row locks the whole time and stalling the scheduler loop.

How to reproduce it (conceptually)

1. Use the FAB auth manager and generate many DAGs from a single DAG file.
2. While the DAG processor is parsing that file (holding SELECT … FOR UPDATE on its dag rows), have the scheduler attempt to update exceeds_max_non_backfill for one of those DAGs.
3. Observe the scheduler statement blocked on Lock/transactionid, and critical_section_duration rising while scheduler CPU falls.

Suggested direction

Reduce the parse transaction's dag-row lock hold and/or the scheduler's contention on it. As a partial step, per-DAG permission sync (another writer in the same parse transaction) can be made to honor [fab] update_fab_perms — proposed in #70589. A more complete fix would shorten/scope the find_orm_dags lock or the scheduler's dag-row write.

Anything else

Related: #23232 (pluggable appless security manager).