Pod is terminated but DAG is still in running state

[shihabcsedu09]

I have configured AKS Kubernetes Cluster where there is an Airflow Scheduler pod and an Airflow Scheduler pod. I have written a DAG that will run using the Kubernetes Pod Operator.

When the DAG is triggered a pod is created and the steps inside the DAG start running. The issue arises when the task is done and the pod moves to a terminated/completed state the airflow webserver seems to not receive that information. For this reason, the DAG status is always running in the webserver.

I observed both the Kubernetes pod logs and the logs that we can see from the web UI for that task. The logs in the webserver are lagging behind from the logs in the pod.

I am using Airflow 1.10.14.

[shihabcsedu09]

Its also happening in Airflow 2.0.1. What I observed is that it happens when there is a long-running airflow step.

[turbaszek]

Please open a bug report. It would be best if you could provide a DAG which can replicate the issue. From what you told it should be simple sleep task.

There are some other issues related to state updated when using k8s operators:
#10325
#11190

[shihabcsedu09]

@turbaszek I have created an issue - 14791