KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods

[mo-rieger]

Under which category would you file this issue?

Providers

Apache Airflow version

2.11.2

What happened and how to reproduce it?

get_pods() issues discover_pods_retry_number + 1 (default 4) list_namespaced_pod calls back to back with no delay, then raises No pods running with labels ... if the pods are not visible yet. Raising discover_pods_retry_number only adds more immediate GETs — there is no way to wait.

Production trace for one occurrence:

time (UTC)	event
00:00:23.223	job.py — Building job job-<task>-<suffix>
00:00:23.360	task failed — AirflowException: No pods running with labels ...
00:00:23	Job controller — SuccessfulCreate: Created pod job-<task>-<suffix>-<id>
00:01:28	Job controller — Job completed

kube_job_created and kube_pod_created are both 00:00:23: the pod existed in the same second the operator gave up, 137 ms after the Job was built. The workload then ran to success unmonitored. ~20 occurrences in 5 days across 8+ unrelated DAGs.

Consequences:

1. The task is marked failed while its Job succeeds. Nothing waits on, monitors or cleans up the Job; on_kill() does not fire.
2. self.pods is never assigned, so _cleanup_monitoring_pods() in the finally block no-ops.
3. A retry or manual clear starts a second Job while the first is still running or has already committed its writes — silent double execution for non-idempotent tasks.

The message is also inaccurate: the pods are not "not running", they do not exist yet.

What you think should happen instead?

Wait for the pods, bounded by a timeout. No new parameters needed — KubernetesJobOperator extends KubernetesPodOperator, so schedule_timeout_seconds (deadline, default 120 via startup_timeout_seconds) and startup_check_interval_seconds (interval, default 5) are already inherited and documented. The first probe stays immediate, so tasks whose pods appear at once are unaffected.

Also: break on >= parallelism rather than ==, and deprecate discover_pods_retry_number as a no-op.

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

cncf provider at 10.13.0 but related code is unchanged in current main

Official Helm Chart version

1.19.0

Kubernetes Version

1.35.6

Helm Chart configuration

Not Applicable

Docker Image customizations

Not Applicable

Anything else?

The issue is Inherited by KubernetesStartKueueJobOperator, GKEStartJobOperator, GKEStartKueueJobOperator — i.e. exactly the Kueue case where pod creation is slowest.

The issue is known and was raised in #53368 but rejected as out of scope for this PR.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct