Dag Processor Liveness Probe Keeps Failing Intermittently

[niraj8241]

Issue with Dag Processor

Environment Details:

Cluster: AWS EKS Running on 1.35
Component: Dag Processor
Number of Dags: 900+
Airflow Version: 3.1.8
Instance Type: r7i.xlarge

Issue Observed: No alive jobs found.

I have recently deployed Airflox 3x. This is running on AWS EKS. The dag process is currently running with 2 replicas with below settings:

dagProcessor:
      enabled: true
      replicas: ${AIRFLOW_DAG_PROCESSOR_REPLICAS:=2}
      podAnnotations:
        ad.datadoghq.com/dag-processor.logs: |
          [ 
            { 
              "service":"airflow-dag-processor",
              "source":"airflow",
              "env":"${AIRFLOW_ENVIRONMENT:=dev}",
            }
          ]
      resources:
        requests:
          cpu: ${AIRFLOW_DAG_PROCESSOR_CPU_REQUEST:=2}
          memory: ${AIRFLOW_DAG_PROCESSOR_MEMORY_REQUEST:=2000Mi}
        limits:
          cpu: ${AIRFLOW_DAG_PROCESSOR_CPU_LIMIT:=3}
          memory: ${AIRFLOW_DAG_PROCESSOR_MEMORY_LIMIT:=4000Mi}
      livenessProbe:
        timeoutSeconds: 60
        failureThreshold: 10
        periodSeconds: 60
      env:
        - name: AIRFLOW__DAG_PROCESSOR__DAG_FILE_PROCESSOR_TIMEOUT
          value: "600"
        - name: AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES
          value: "4"
        - name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
          value: "180"
        - name: AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE
          value: "random_seeded_by_host"
        - name: AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL
          value: "900"
        - name: AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD
          value: "600"

Every now and then the Dag Processor just keeping passing and failing causing the new dags and update to existing dags be not updated for several hours.

-- Snippet of Restarts --

Observation:

I do see that the dag processor is able to see the files in my dags folder but then it does not processor them. During dags update when I go inside the container and see the files, the files are updated files but when the jobs run on Airflow, it refers to the older version of the file which earlier had issues. So I have a feeling that the code is never registered or acknowledged by Airflow.

dags-folder  classic/processor.py                                                                                                                4627  252.88s                    0           0
================================================================================ [airflow.dag_processing.manager.DagFileProcessorManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:815
2026-06-05T17:41:28.823070Z [info     ] **Heartbeat recovered after 240.17 seconds** [airflow.jobs.job.Job] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=job.py:253
2026-06-05T17:41:28.833219Z [info     ] Not time to refresh bundle dags-folder [airflow.dag_processing.manager.DagFileProcessorManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:585
2026-06-05T17:41:30.248570Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:179
2026-06-05T17:41:30.277134Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:179
2026-06-05T17:41:33.281455Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:179
2026-06-05T17:41:33.297554Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:179
2026-06-05T17:41:34.403258Z [info     ] Not time to refresh bundle dags-folder [airflow.dag_processing.manager.DagFileProcessorManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:585
2026-06-05T17:41:34.685971Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_processing.bundles.manager.DagBundlesManager] dd.env= dd.service=airflow-dag-processor dd.span_id=0 dd.trace_id=0 dd.version= loc=manager.py:179
2026-06-05T17:41:36.232343Z [info     ] DAG bundles loaded: dags-folder [airflow.dag_p

Any help on fixing this would be greatly appreciated. If I am missing any particular piece of information, please let me know.

[bkarankar]

it's may be due to too many dags or resources issue, share more info by running below commands

airflow dags report
airflow dags list-import-errors

or, you can let's try this,
update variables and run on ubuntu 24.04. it has all pre-configured settings with azure sso and airflow api/cli enbled. make sure to update the password and variables.

https://github.com/bkarankar/Nexoryx_Airflow3