Airflow stops scheduling dagruns for other dags if stale dags are not deactivated

[deepak4babu]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.0.6

What happened?

When stale dags are not deactivated, then airflow scheduler gives errors like below. Eventually, the number of dagruns created by airflow is reduced and the overall throughput decreases.

What you think should happen instead?

Airflow scheduler should ignore the dags that is not in serialized table and create dagruns for others.

How to reproduce

Create a 2.x airflow environment with 10+ dags. Make sure the multiple dagruns can run concurrently for all dags. Upgrade airflow to 3.0.6 and at the same time remove atleast 5 dags during the upgrade.
Airflow will eventually stops scheduling dagruns for the other dags

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-imap==3.9.2
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-standard==1.6.0

Deployment

Other

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Hrishikesh19032004]

Hello! I’d like to work on this issue. I’ve cloned the Airflow repository and set up the project locally, and I’m currently working on reproducing the problem and investigating the scheduler behavior around stale DAGs. Can you assign this issue to me.

[Arunodoy18]

I investigated and the scheduler is attempting to create DagRuns for stale SerializedDagModel entries when DAG files were removed during upgrade to Airflow 3. This blocks scheduling throughput for valid DAGs.

I plan to fix this by filtering serialized DAGs against the current DagBag before DagRun creation, ensuring removed DAGs are skipped safely. I will also add a unit test reproducing the stale serialized DAG scenario.