Is HA mode for airflow dag-processor supported?

[bmoon4]

Hi all,

I am wondering whether running multiple airflow dag-processor instances (pods) for high availability is currently supported.

If this isn't currently supported, is HA for the dag-processor on the roadmap?

Thanks :)

[apoorva-01]

Short version: there's no active/standby HA for the dag-processor, and running two of them against the same DAGs isn't a redundancy setup, it's just duplicated parsing work. Two processors watching the same bundle both parse everything, and historically they'd even fight over deactivating each other's DAGs. The "make it horizontally scalable" ask is tracked in #32966, still open, and the PR that tried it got closed without merging. So no, not on the roadmap in the shipped-and-supported sense today.

What you can do in Airflow 3 is shard by bundle. Split your DAGs into multiple bundles and run one processor per set:

airflow dag-processor -B bundle-a -B bundle-b
airflow dag-processor -B bundle-c

-B/--bundle-name can be passed more than once, so each processor owns a disjoint slice. That scales parsing horizontally, it just doesn't give you a hot standby for any single bundle.

For actual availability the practical answer is the boring one: run it under something that restarts it. On K8s a single-replica Deployment (or the official Helm chart's dagProcessor) will reschedule the pod if it dies. You lose parsing for the few seconds it takes to come back, but nothing gets scheduled off stale DAGs in the meantime, so it's usually fine. Just don't set that replica count to 2 on the same bundle expecting failover, you'll get double parsing, not redundancy.