Skip to content

Fix KubernetesPodTrigger pod terminal state handling#66650

Merged
jscheffl merged 3 commits into
apache:mainfrom
anmolxlight:fix/kubernetes-pod-trigger-pod-state
May 10, 2026
Merged

Fix KubernetesPodTrigger pod terminal state handling#66650
jscheffl merged 3 commits into
apache:mainfrom
anmolxlight:fix/kubernetes-pod-trigger-pod-state

Conversation

@anmolxlight
Copy link
Copy Markdown
Contributor

Summary

  • Add pod-level terminal-state detection to KubernetesPodTrigger
  • Use pod phase to stop polling when Kubernetes marks the pod as Failed or Succeeded
  • Keep container-state fallback for non-terminal pod phases
  • Update deferrable operator pre-deferral terminal-state check to use pod state

Tests

  • uv run --project providers/cncf/kubernetes pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py -q
  • uv run --project providers/cncf/kubernetes pytest --with-db-init providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py::TestKubernetesPodOperatorAsync::test_async_create_pod_should_execute_successfully -q

Closes #66624

@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels May 10, 2026
Comment thread providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Kubernetes deferrable pod trigger/operator flow to stop polling based on pod-level terminal phases (Succeeded/Failed) instead of relying solely on the base container state, addressing cases like init-container failure where the pod is already terminal but the base container never transitions.

Changes:

  • Add define_pod_state() in KubernetesPodTrigger to treat pod Succeeded/Failed as terminal outcomes, falling back to base-container state otherwise.
  • Switch trigger polling and operator pre-deferral terminal-state checks to use define_pod_state().
  • Update/extend unit tests to patch the new method and add coverage for the init-container failure scenario.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py Introduces pod-phase terminal detection and uses it in trigger polling.
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py Uses pod-level terminal detection to skip deferral when already terminal.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py Updates mocks to define_pod_state() and adds a regression test for pod failed while base container waits.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py Updates operator async test to patch define_pod_state() instead of define_container_state().
Comments suppressed due to low confidence (1)

providers/cncf/kubernetes/tests/unit/cncf/kubernetes/triggers/test_pod.py:336

  • test_running_log_interval patches define_pod_state, but the injected mock argument is still named define_container_state. Renaming the parameter (and any related variable names) to match the patched symbol would avoid confusion and make the test intent clearer.
    @mock.patch("airflow.providers.cncf.kubernetes.triggers.pod.datetime")
    @mock.patch(f"{TRIGGER_PATH}.define_pod_state")
    @mock.patch(f"{TRIGGER_PATH}._wait_for_pod_start")
    @mock.patch(
        "airflow.providers.cncf.kubernetes.triggers.pod.AsyncPodManager.fetch_container_logs_before_current_sec"
    )
    @mock.patch("airflow.providers.cncf.kubernetes.triggers.pod.AsyncKubernetesHook.get_pod")
    async def test_running_log_interval(
        self,
        mock_get_pod,
        mock_fetch_container_logs_before_current_sec,
        mock_wait_pod,
        define_container_state,
        mock_datetime,

Comment thread providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/triggers/pod.py Outdated
@jscheffl jscheffl merged commit bccbcbf into apache:main May 10, 2026
112 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KubernetesPodTrigger runs forever if non-base container in pod fails

3 participants