dagProcessor liveness probe always fails

[IanSweeneyKodiak]

Under which category would you file this issue?

Helm chart

Apache Airflow version

3.2.2 (helm 1.2.2)

What happened and how to reproduce it?

Default helm chart uses livenessProbe with command: ~ for dagProcessor.
This uses the default command to check liveness which always fails:
airflow jobs check --job-type DagProcessorJob --hostname $(hostname)
Causing the dagProcessor to be restarted by k8s every 5min (livenessProbe failure)

What you think should happen instead?

No response

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

1.22.0 (latest released)

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[IanSweeneyKodiak]

Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs
select * from job where job_type='DagProcessorJob' order by start_date desc;
206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow

airflow jobs check --job-type DagProcessorJob
No alive jobs found

It seems that maybe the dagProcessor is not sending a heartbeat?

Guessing this is because the heartbeat is only_if_necessary

airflow/airflow-core/src/airflow/jobs/dag_processor_job_runner.py

Line 55 in ba8cfe4

only_if_necessary=True,

Maybe due to the config option
scheduler.job_heartbeat_sec: 30

[Miretpl]

If you don't see missing dags or in the dag-processor logs, you see that the dag-processor is working, I would assume that there is high CPU usage + maybe some bottlenecks on the database, and the healthcheck command just passes the execution timeout. If that is the case, you can modify resources, maybe optimise Dags code regarding parsing and execution layers (https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code, which in general should decrease workload on the Dag Processor) and tweak a little Dag Processor configuration like parsing processor, etc.

From more fixes like workarounds, there are some options regarding the probe itself within the chart: https://github.com/apache/airflow/blob/helm-chart/1.22.0/chart/values.yaml#L3052.

[tirkarthi]

What is the value of hostname command output inside the pod? Does the value match the entries found in the database with heartbeat timestamp updated?

[arkadiuszbach]

Turns out there is a running dagProcessor job in the db but the cli used for the health check returns no alive jobs select * from job where job_type='DagProcessorJob' order by start_date desc; 206 | | running | DagProcessorJob | 2026-06-26 17:33:29.530223+00 | | 2026-06-26 17:37:04.932384+00 | | airflow-dag-processor-5df45489f7-92r7z | airflow

airflow jobs check --job-type DagProcessorJob No alive jobs found

It seems that maybe the dagProcessor is not sending a heartbeat?

Guessing this is because the heartbeat is only_if_necessary

airflow/airflow-core/src/airflow/jobs/dag_processor_job_runner.py

Line 55 in ba8cfe4

only_if_necessary=True,
Maybe due to the config option scheduler.job_heartbeat_sec: 30

Yes its because of scheduler.job_heartbeat_sec: 30 and dag_processor.health_check_threshold: 30, see: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#health-check-threshold

just increase health_check_threshold for the dag_processor

[eajimmy]

Same problem here on Airflow 3.2.1 (KubernetesExecutor + CeleryExecutor).
I confirmed it with this query, running it repeatedly:

SELECT id, hostname, state,
       latest_heartbeat,
       EXTRACT(EPOCH FROM (now() - latest_heartbeat)) AS age_seconds
FROM job
WHERE job_type = 'DagProcessorJob'
ORDER BY latest_heartbeat DESC
LIMIT 30;

age_seconds oscillates continuously between ~30 and ~60s and never drops close to 0.

Setting AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 fixes it, as mentioned above.

I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.

[IanSweeneyKodiak]

Would it be possible/correct to update this default in the helm chart then?
AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120

[Miretpl]

You can adjust its configuration in the values.yaml file in the config section or by adding environment variables in extraEnv or envs at the component level. Looking at the current state of things, I would go with config or extraEnv.

[Miretpl]

I also tried increasing parsing_processes and min_file_process_interval, but these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to parsing load.
I even confirmed this with the processor fully idle (min_file_process_interval=300, no files to parse) and the age still stayed in the 30–60s range.

This indicates insufficient resources on the database, such as CPU or IOPS.

Would it be possible/correct to update this default in the Helm chart then?
AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120

To be honest, I don't see the reason to do that. At least for now, there is a lack of information regarding particular components' configurations and resources, as well as the dags' characteristics themselves, which are impacting the load on both the dag processor and the database. From my experience with quite big Airflow environments (comparing, e.g. to the data from the Airflow Survey), when the database is correctly configured with quality gates for dags code and some tweaks to dag processor configuration (depending on the dags characteristics), the default settings are sufficient. Also, adjusting this configuration is easy, and I would say not problematic. If adjusting it wasn't possible, it would be a totally different story.

I will convert this issue to a discussion since we don't have enough data to determine whether it is environment-specific or an Apache Airflow issue.