Skip to content

Add KubernetesPodExecOperator for existing Kubernetes Pods - #71244

Open
AlejandroMorgante wants to merge 4 commits into
apache:mainfrom
AlejandroMorgante:add-kubernetes-pod-exec-operator
Open

Add KubernetesPodExecOperator for existing Kubernetes Pods#71244
AlejandroMorgante wants to merge 4 commits into
apache:mainfrom
AlejandroMorgante:add-kubernetes-pod-exec-operator

Conversation

@AlejandroMorgante

Copy link
Copy Markdown
Contributor

Add KubernetesPodExecOperator to execute commands in a running container of an existing Kubernetes Pod without creating, restarting, or deleting it. This supports pre-warmed and API-visible static Pods where avoiding startup latency is important.

The operator validates the target Pod and container, streams stdout and stderr, supports optional stdout XCom, reports non-zero exit codes, and closes the exec connection when the task is killed. Provider metadata, RBAC guidance, and operator documentation are included.

Tests:

  • pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod_exec.py -q (32 passed)
  • Provider mypy check
  • Provider metadata and documentation validation
  • Pre-commit and manual checks

Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

@boring-cyborg boring-cyborg Bot added area:providers kind:documentation provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Aug 6, 2026
@AlejandroMorgante AlejandroMorgante changed the title Add KubernetesPodExecOperator to execute commands in existing Kubernetes Pods Add KubernetesPodExecOperator for existing Kubernetes Pods Aug 6, 2026
@AlejandroMorgante
AlejandroMorgante marked this pull request as draft August 6, 2026 15:40
@AlejandroMorgante
AlejandroMorgante marked this pull request as ready for review August 6, 2026 15:40
@AlejandroMorgante
AlejandroMorgante marked this pull request as draft August 6, 2026 17:41
@AlejandroMorgante
AlejandroMorgante marked this pull request as ready for review August 6, 2026 17:41
@AlejandroMorgante
AlejandroMorgante force-pushed the add-kubernetes-pod-exec-operator branch from 031ffbc to 52b1cc6 Compare August 6, 2026 21:59
@kaxil

kaxil commented Aug 7, 2026

Copy link
Copy Markdown
Member

Add KubernetesPodExecOperator to execute commands in a running container of an existing Kubernetes Pod without creating, restarting, or deleting it. This supports pre-warmed and API-visible static Pods where avoiding startup latency is important.

The operator validates the target Pod and container, streams stdout and stderr, supports optional stdout XCom, reports non-zero exit codes, and closes the exec connection when the task is killed. Provider metadata, RBAC guidance, and operator documentation are included.

Tests:

  • pytest providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod_exec.py -q (32 passed)
  • Provider mypy check
  • Provider metadata and documentation validation
  • Pre-commit and manual checks
Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

What's the User story and real-world use cases for this?

@AlejandroMorgante

Copy link
Copy Markdown
Contributor Author

@kaxil

What's the User story and real-world use cases for this?

Thanks for asking. The user story is:

As an Airflow user who already manages a long-running Kubernetes Pod, I want to execute commands in one
of its containers as Airflow tasks, without Airflow creating, restarting, or deleting that Pod.

Our concrete use case is running dbt on EKS. Today, our workflow starts a new Pod, initializes the dbt project and its dependencies, and then executes a command such as dbt run. A significant part of the execution time is spent scheduling the Pod and initializing the environment.

We want to manage that Pod separately and keep it warm, then use Airflow to execute ad hoc commands such as dbt run, dbt test, or dbt build. This avoids the repeated cold-start and initialization cost.

More generally, this is useful whenever a Pod’s lifecycle is managed outside the task and its environment is already initialized. The operator exposes the equivalent of kubectl exec as an Airflow task, including Airflow logging and optional XCom output, without assuming ownership of the Pod lifecycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants