Scheduler failing to adopt pods with code 409 (Conflict)

[pardini01]

Kubernetes version: v1.23 (EKS)
Airflow version: 2.4.3

After a few seconds from the start of a DAG with a large number of parallel tasks (+100), the Airflow scheduler starts trying to adopt lots of those tasks, probably because those were triggered by the "mini-scheduler":

[2022-12-07 12:59:26,177] {kubernetes_executor.py:730} INFO - attempting to adopt pod podname-000571bffc6d4fa28eb9bec091dc149a
[2022-12-07 12:59:26,208] {kubernetes_executor.py:730} INFO - attempting to adopt pod podname-01d660ad058247e2a1bc4e70987079d3
[2022-12-07 12:59:26,237] {kubernetes_executor.py:730} INFO - attempting to adopt pod podname-04e1b2ac85f64c698f8e12951f38c987
...

It then fails to adopt a few with a 409 (Conflict) error

[2022-12-07 12:59:26,348] {kubernetes_executor.py:746} INFO - Failed to adopt pod podname-0bc5e65f42b64cf0adb93b85d59d81cb. Reason: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Audit-Id': '488e68f5-5d5a-48a5-b717-8ca0983d62dc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '3c24a4a4-2ab0-44e1-bf6c-ad7823e8e769', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'a03f01b4-157d-4d40-80a1-e012c555e241', 'Date': 'Wed, 07 Dec 2022 12:59:26 GMT', 'Content-Length': '410'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on pods \"podname-0bc5e65f42b64cf0adb93b85d59d81cb\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":{"name":"podname-0bc5e65f42b64cf0adb93b85d59d81cb","kind":"pods"},"code":409}
...

and finally, a subset (but usually all) of the tasks that became orphans and were reset by the scheduler receive a SIGTERM, as the following task log shows:

[2022-12-07, 13:00:48 UTC] {local_task_job.py:222} WARNING - State of this instance has been externally set to failed. Terminating instance.
...
[2022-12-07, 13:00:48 UTC] {taskinstance.py:1562} ERROR - Received SIGTERM. Terminating subprocesses.

This used to be more frequent when the Airflow scheduler was restarted due to other issues and it had to readopt all its tasks. After we handled those restarts, it became less frequent.

Unfortunately, this is not easily reproducible, that's why I didn't submit an Issue, but it is frequent enough to bug us with the time wasted in the retry interval, and also the interruption of the computation.

It is worth mentioning that we run several Airflow instances in the same Kubernetes cluster and namespace, which is giving us the idea that the other schedulers might be interacting with the Pods and changing them.

[potiuk]

Mini schedulers do not start new pods. I think your "un several Airflow instances in the same Kubernetes cluster and namespace" statement explains it well. DON'T DO IT. It's what causes the problem and the idea of doing it makes very little sense and is not supported by Airflow..

[pardini01]

Thank you for the reply! Is there any documentation about this?

[potiuk]

Is there any documentation you can do it ?

[pardini01]

I don't think so, but it worked for us for a long time before this issue started. I feel like this should be in a "best practices" doc. Anyway, we will start working on using separate namespaces. Thanks!

[potiuk]

We are learning from our users every day something new. And things that seem obvious for some of the maintainers (like that one) might not be as obvious for our users. That's why Airflow (open source project created by > 2300 contributors as of today) is open to any improvements in the docs that would make it clearer for them. If you think there is a place that we could add such clarification (best if you can confirm that it solves your problem) then adding it is the smallest thing to do to give back for the software that you get for free. Absolutely free.

So feel free to add some documentation about it when you confirm it works - anyone can suggest a change by simply finding the right page to add the information and clicing "Suggest a change on this page" button at the right-bottom corner and PR will be opened. As mentioned above Airflow has > 2300 and many of them contributed such fixes to docs for example. you can become one one of those contributors by doing so and start giving back to the community.

And you are probably at this moment the best person in the world to know where to add it and how to phrase it - I guess when you made a decision about using such an approach you looked in the docs (otherwise you wouldn't have asked). So where and how to add it so that other people like you would find it when they are looking for it is probably somethign you know best :).

Happy to review it when you submit it.

[pardini01]

For sure I would be glad to submit this contribution once we confirm that our side is fixed and also would appreciate your review 😃