Dag with kubernetes executor are being killed with "PID of job runner does not match"

[ut0mt8]

Apache Airflow version

2.5.1

What happened

I think this issue have been already discussed in many issue but I'm still in the case where I basically cannot use the kubernetes Executor...

with this config:

   executor = KubernetesExecutor

[scheduler]
    job_heartbeat_sec = 60
    schedule_after_task_execution = False
    scheduler_health_check_threshold = 30
    scheduler_heartbeat_sec = 10

and this very simple Dag

import datetime

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
    

def sigterm_debug(ds, **kwargs):
    run_date = str(ds)
    print("run date is : " + str(run_date))
    import time
    time.sleep(500)
    print("yeaah not killed ...")


dag = DAG(
    dag_id="debug",
    description="Debug",
    schedule_interval="30 21,9 * * *",
    start_date=datetime.datetime(2023, 1, 1),
    catchup=False,
    dagrun_timeout=datetime.timedelta(hours=3)
)


sigterm_debug_task = PythonOperator(task_id='sigterm_debug_task', 
                                    python_callable=sigterm_debug,
                                    execution_timeout=datetime.timedelta(hours=1),
                                    retries=2, 
                                    dag=dag)


sigterm_debug_task

the Dag is killed by the scheduler raising the infamous exception: "PID of job runner does not match" at the exact time the job_heartbeat_interval run

Note: the pod by itself is fine. This is really airflow that kill the task and then the pod exit with an exit code > 0...

Note2: increasing "job_heartbeat_sec" to something very high let the DAG finish successfully but then the pods is never ripped before the job_heartbeat_sec arise ...

What you think should happen instead

Everything should be working as expected :)

How to reproduce

With the latest version, my conf and this simple dag.

Operating System

Docker/Kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

100% reproducible with this config.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ut0mt8]

@potiuk did that mean that is an know bug? it's very very annoying and make the kube executor quasi unusable.

[potiuk]

I have no idea, but I doubt. I added "needs-triage" (our new approach to mark issues as something that someone needs to have a look at) and possibly someone will take a look at it. I think adding milestone was just a mistake.

[ut0mt8]

btw I ended monkey patching airflow

airflow/airflow/jobs/local_task_job.py

Line 270 in 2cd12fc

raise AirflowException("PID of job runner does not match")

removing this line...
works for me.

[jedcunningham]

Hi @ut0mt8,

I naively took your DAG and plopped it into my local env with this config:

$ cat 29389.yaml                    
dags:
  persistence:
    enabled: true
    existingClaim: {my dags mount}
                                 
executor: KubernetesExecutor

config:
  scheduler:
    job_heartbeat_sec: 60
    schedule_after_task_execution: False
    scheduler_health_check_threshold: 30
    scheduler_heartbeat_sec: 10

(and just to record it here for the future, the latest OSS chart uses 2.5.1 as of today).

Unfortunately, I wasn't able to produce your issue. I'm not really sure what might be causing it, but could you put together a full example so we can reproduce this ourselves? DAG, Dockerfile (it'd make getting the dag in easier, and more consistent between our envs), helm values files?

Thanks!