Add `--min-completed-minutes` to `cleanup-pods` to prevent KPO race condition by noamst-monday · Pull Request #70595 · apache/airflow

noamst-monday · 2026-07-28T11:48:25Z

Problem

KubernetesPodOperator in synchronous (deferrable=False) mode calls PodManager.await_pod_completion to confirm the pod reached a terminal phase. This method polls read_pod every 2 seconds (pod_manager.py:839, hardcoded time.sleep(2)).

When a pod transitions to Succeeded, there is up to a 2-second window before PodManager.await_pod_completion observes the terminal phase. The airflow kubernetes cleanup-pods command currently deletes Succeeded/Failed/Evicted pods immediately — there is no minimum-age guard for terminal states.

If the cleanup job fires during that window, PodManager.await_pod_completion's next read_pod call returns 404 and the task is marked FAILED, even though the pod completed successfully (exit code 0).

A --min-pending-minutes guard already exists for Pending pods (default 30 m, minimum 5 m). No equivalent exists for terminal states.

Real-world reproduction

This was confirmed via Kubernetes API server audit logs on a production EKS cluster running apache-airflow-providers-cncf-kubernetes==10.19.0 on Airflow 3.2.2. The cleanup CronJob was set to run every 5 minutes.

Timeline of a representative failure:

13:15:12Z  PodManager polls pod       → phase Running
13:15:14Z  Container exits            → exit code 0 (pod phase → Succeeded)
13:15:17Z  cleanup-pods deletes pod   → 3 s after completion
13:15:18Z  PodManager polls pod       → 404 Not Found → task FAILED

The pod had succeeded; the task failure was a false positive. Multiple DAGs exhibited the same pattern on the same day.

Airflow task log excerpt:

[INFO]  Pod run-o1sxc2on has phase Running
[ERROR] Task failed with exception
ApiException: (404) Reason: Not Found
HTTP response body: {"message":"pods \"run-o1sxc2on\" not found","reason":"NotFound","code":404}

Audit log at deletion time confirmed phase: Succeeded, containerStatuses[0].state.terminated.exitCode: 0, finishedAt: 13:15:14Z.

Reducing the CronJob frequency is a partial mitigation (lowers probability) but does not eliminate the race — the window is a function of the poll interval, not the CronJob frequency.

Solution

Add --min-completed-minutes (default 0, preserving existing behaviour) to the cleanup-pods command. When set to any positive value, Succeeded/Failed/Evicted pods are skipped unless their completion time exceeds the threshold.

Completion time is derived from max(containerStatuses[*].state.terminated.finishedAt) — the latest container finish time across all containers — which is the most accurate signal for "when the pod entered a terminal state". Falls back to metadata.creationTimestamp for pods evicted before any container started.

Setting --min-completed-minutes=5 gives PodManager.await_pod_completion a 5-minute observation window — 150× wider than the 2 s poll interval — closing the race completely in practice.

Changes

File	Change
`cli/definition.py`	Add `ARG_MIN_COMPLETED_MINUTES` (default `0`, `allow_zero=True`); wire into cleanup-pods args
`cli/kubernetes_command.py`	Add `_get_pod_completion_time()` helper; gate terminal-pod deletion by age
`tests/.../test_kubernetes_command.py`	4 new unit tests (too-young Succeeded, old-enough Succeeded, default=0 backward compat, too-young Failed/Never)
`docs/changelog.rst`	Entry under 10.21.0

CLI reference docs (cli-ref.rst) use .. argparse:: and pick up the new flag automatically.

Testing

pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/cli/test_kubernetes_command.py::TestCleanUpPodsCommand -v

All existing tests continue to pass (default=0 preserves current behaviour). Four new tests cover the new flag.

Usage

In the cleanup CronJob, add the flag:

airflow kubernetes cleanup-pods --namespace airflow --min-completed-minutes 5

Or in the Airflow Helm chart values:

cleanup:
  args: ["bash", "-c", "exec airflow kubernetes cleanup-pods --namespace {{ .Release.Namespace }} --min-completed-minutes 5"]

Was generative AI tooling used to co-author this PR?

Yes (Claude Sonnet 4.6)

Generated-by: Claude Sonnet 4.6 following the guidelines

Read the Pull Request Guidelines for more information.

…tion The ``airflow kubernetes cleanup-pods`` command currently deletes Succeeded/Failed/Evicted pods immediately, with no minimum-age guard for terminal states. ``KubernetesPodOperator`` in synchronous mode polls pod status every ~2 seconds via ``await_pod_completion``. If the cleanup job fires in the window between the pod reaching ``Succeeded`` and KPO's next poll, KPO receives a 404 and fails the task -- even though the pod completed successfully (exit code 0). A ``--min-pending-minutes`` guard already exists for Pending pods (default 30 m, floor 5 m). No equivalent exists for terminal states. This commit adds ``--min-completed-minutes`` (default ``0``, which preserves the existing behaviour). When set to any positive value, Succeeded/Failed/Evicted pods are skipped unless their completion time is older than the threshold. Completion time is derived from the latest ``containerStatuses[*].state.terminated.finishedAt`` timestamp (falls back to ``metadata.creationTimestamp`` for pods that were evicted before any container started). Root-cause investigation ------------------------ This was confirmed via Kubernetes API server audit logs on a production EKS cluster. Timeline for an affected KPO task: 13:15:12Z KPO polls pod → phase Running 13:15:14Z Container exits with code 0 (pod transitions to Succeeded) 13:15:17Z cleanup-pods CronJob deletes the pod (3 s after completion) 13:15:18Z KPO polls pod → 404 Not Found → task marked FAILED The pod had succeeded; the task failure was a false positive caused entirely by the race. Multiple production DAGs exhibited the same pattern with the cleanup CronJob set to run every 5 minutes. Reducing the CronJob frequency is a partial mitigation (lowers the probability) but does not eliminate the race. Setting ``--min-completed-minutes=5`` gives KPO a 5-minute window to observe the terminal phase -- 150x wider than the 2 s poll interval -- closing the race completely in practice. Changes ------- * ``definition.py`` – add ``ARG_MIN_COMPLETED_MINUTES``; wire into cleanup-pods args tuple * ``kubernetes_command.py`` – add ``_get_pod_completion_time()`` helper; gate terminal-pod deletion by age when ``min_completed_minutes > 0`` * ``test_kubernetes_command.py`` – 4 new unit tests covering: - Succeeded pod too young → not deleted - Succeeded pod old enough → deleted - Default (0) preserves immediate-deletion behaviour - Failed/Never pod too young → not deleted * ``changelog.rst`` – entry under 10.21.0 CLI docs (``cli-ref.rst``) are auto-generated via ``.. argparse::`` and will pick up the new flag automatically.

boring-cyborg · 2026-07-28T11:48:32Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

noamst-monday · 2026-07-28T14:41:31Z


+ARG_MIN_COMPLETED_MINUTES = Arg(
+    ("--min-completed-minutes",),
+    default=0,


Do we want a different default here? Maybe allow >3 seconds grace by default to be safe from KPO polling interval?

noamst-monday · 2026-07-28T14:41:54Z

+    for status in pod.status.container_statuses or []:
+        if status.state and status.state.terminated and status.state.terminated.finished_at:
+            times.append(status.state.terminated.finished_at)
+    return max(times) if times else pod.metadata.creation_timestamp


Not sure about this one, open for suggestions

jedcunningham · 2026-07-28T14:23:40Z

+    type=positive_int(allow_zero=True),
+    help=(
+        "Minimum age in minutes of a completed (Succeeded/Failed/Evicted) pod before it is deleted. "
+        "Defaults to 0 (delete immediately, preserving current behaviour). "


Lets set a default age to avoid this race. No reason to maintain a race condition :)

jedcunningham · 2026-07-28T14:24:49Z

+        "Set this to a positive value to prevent a race condition where the cleanup job removes a "
+        "just-completed pod before KubernetesPodOperator has polled its terminal phase, "
+        "causing a spurious task failure despite the pod having succeeded."


We can probably shorten or remove this completely.

jedcunningham · 2026-07-28T15:02:02Z

+    the time the *last* container finished, not by when the pod was created.
+    """
+    times = []
+    for status in pod.status.container_statuses or []:


Init container statuses live in a separate field, pod.status.init_container_statuses, so they're never seen.

That matters because of the fallback direction. When an init container fails, the pod goes Failed/restartPolicy: Never, but the base container never starts, so it has no terminated state and times comes back empty. Then we return creation_timestamp, which is always earlier than real completion, so the age comes out inflated and the pod is deleted immediately. Same for pods evicted before containers started. In other words the guard silently no-ops in some of the exact terminal states the flag advertises, and it fails in the unsafe direction (delete too early) rather than the safe one.

Two changes:

Scan init container statuses too:

statuses = [*(pod.status.container_statuses or []), *(pod.status.init_container_statuses or [])]

Pick a fallback that isn't strictly older than completion. max(c.last_transition_time for c in pod.status.conditions) is at-or-after the terminal transition (the Ready condition flips on eviction) and is always present for a scheduled pod.

Also, no test currently covers either behavior the helper exists for — every case builds exactly one container status, so max() over multiple is unexercised, and because _make_pod uses a bare MagicMock(), status.state and status.state.terminated are auto-truthy and the None guards can't be exercised as false. A small parametrized unit test on _get_pod_completion_time with real k8s.V1ContainerStatus objects would cover both plus the fallback. Or, cover it with more comprehensive coverage of the cli command directly would work too.

Thanks! Appreciate your review, I'll take a look and update.

noamst-monday requested review from hussein-awala, jedcunningham and jscheffl as code owners July 28, 2026 11:48

boring-cyborg Bot added area:providers kind:documentation provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jul 28, 2026

noamst-monday added 2 commits July 28, 2026 14:48

Update changelog with PR number apache#70595

9a1de45

Fix ruff D213 docstring format in _get_pod_completion_time

2df806e

noamst-monday marked this pull request as draft July 28, 2026 11:52

noamst-monday added 2 commits July 28, 2026 14:56

Use exact poll interval (2 s) in --min-completed-minutes help text

c32ebfb

Improve --min-completed-minutes help text clarity

90e28ce

noamst-monday marked this pull request as ready for review July 28, 2026 13:24

Merge branch 'main' into add-min-completed-minutes-cleanup-pods

dafde57

eladkal reviewed Jul 28, 2026

View reviewed changes

Comment thread providers/cncf/kubernetes/docs/changelog.rst

noamst-monday commented Jul 28, 2026

View reviewed changes

Revert changelog.rst — auto-generated by release manager

098f5fe

jedcunningham requested changes Jul 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `--min-completed-minutes` to `cleanup-pods` to prevent KPO race condition - #70595

Add `--min-completed-minutes` to `cleanup-pods` to prevent KPO race condition#70595
noamst-monday wants to merge 7 commits into
apache:mainfrom
noamst-monday:add-min-completed-minutes-cleanup-pods

noamst-monday commented Jul 28, 2026 •

edited

Loading

Uh oh!

boring-cyborg Bot commented Jul 28, 2026

Uh oh!

Uh oh!

noamst-monday Jul 28, 2026

Uh oh!

noamst-monday Jul 28, 2026 •

edited

Loading

Uh oh!

jedcunningham Jul 28, 2026

Uh oh!

jedcunningham Jul 28, 2026

Uh oh!

jedcunningham Jul 28, 2026

Uh oh!

noamst-monday Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

noamst-monday commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Real-world reproduction

Related

Solution

Changes

Testing

Usage

Was generative AI tooling used to co-author this PR?

Uh oh!

boring-cyborg Bot commented Jul 28, 2026

Uh oh!

Uh oh!

noamst-monday Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

noamst-monday Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jedcunningham Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

jedcunningham Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

jedcunningham Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

noamst-monday Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

noamst-monday commented Jul 28, 2026 •

edited

Loading

noamst-monday Jul 28, 2026 •

edited

Loading