Skip to content

Mark KubernetesPodOperator and AgentOperator as durable capable - #70289

Merged
amoghrajesh merged 3 commits into
apache:mainfrom
astronomer:mark-kpo-and-agent-operator-as-durable
Jul 24, 2026
Merged

Mark KubernetesPodOperator and AgentOperator as durable capable#70289
amoghrajesh merged 3 commits into
apache:mainfrom
astronomer:mark-kpo-and-agent-operator-as-durable

Conversation

@amoghrajesh

@amoghrajesh amoghrajesh commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes claude sonnet

What

Enhancement to: #69651, where the Airflow Registry added a supports_durable_execution signal (built on top of ResumableJobMixin), badging operators that survive a worker crash/retry without redoing expensive work. KubernetesPodOperator (#69914) and AgentOperator (common.ai) both already implement this same crash-safe property, but neither inherits ResumableJobMixin -- they implement it directly against task_state_store, since their own execution shapes (pod lifecycle management, LLM step/tool-call replay caching) don't fit the mixin's submit/poll/result contract.

The registry detection is purely MRO-based, so both operators are currently invisible to it despite genuinely qualifying.

Proposed change

Adds __supports_durable_execution: ClassVar[bool] = True directly to both operator classes, indicating that durable execution is implemented directly here, not via ResumableJobMixin. This is a plain class attribute, not a new mixin or shared marker class -- there's no common contract between pod-lifecycle management and LLM-call caching to anchor a reusable base class on, so a class-level declaration is the simplest correct signal.

The name uses a double leading underscore deliberately, not a single one. Python name-mangles it to _KubernetesPodOperator__supports_durable_execution / _AgentOperator__supports_durable_execution on each declaring class. This matters because both operators have subclasses that override execute() themselves (e.g. SparkKubernetesOperator, KubernetesJobOperator, KubernetesStartKueueJobOperator for KPO) -- we have no way to verify those subclasses preserve the task_state_store reconnect behavior, so the attribute must not be treated as true for them just because they inherit from a durable-capable parent. A subclass that doesn't redeclare the attribute in its own class body will not match a per-class mangled-name lookup, unlike a plain single-underscore attribute, which would be silently inherited by every subclass regardless of whether it actually preserves the mechanism.

Changes of Note

This attribute is not read by anything yet. It will be a follow-up change, not part of this PR.

What's next

A follow-up change to dev/registry/extract_parameters.py's is_durable_capable() to also treat __supports_durable_execution = True as qualifying. The check must compute the mangled attribute name per class being inspected (f"_{cls.__name__}__supports_durable_execution") rather than a fixed string, so it only matches a class that declares the attribute itself, not one that merely inherits from a durable-capable base.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py Outdated
Comment thread providers/common/ai/tests/unit/common/ai/operators/test_agent.py Outdated
@amoghrajesh
amoghrajesh merged commit f736077 into apache:main Jul 24, 2026
108 checks passed
@amoghrajesh
amoghrajesh deleted the mark-kpo-and-agent-operator-as-durable branch July 24, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Development

Successfully merging this pull request may close these issues.

3 participants