You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Users browsing the Airflow Registry cannot tell which operators support resumption -- i.e., operators that can pick up from where they left off after a worker crash, restart, or retry, rather than re-running from scratch.
This matters for users planning long-running or expensive workloads (data transfers, ML training, large queries) where re-running from scratch on retry is wasteful or unsafe.
Proposal
Add a visible indicator on operator pages (and a filter on listing pages) in the Airflow Registry that marks operators as resumable.
Open questions to resolve in this issue:
Definition. What qualifies as "resumable"? Candidates: deferrable operators (trigger-based resumption), operators with explicit checkpointing, idempotent transfer operators that can resume from offset/cursor, or some combination. Pick one definition or model it as multiple flags.
Source of truth. How do we get this signal? Options: a class attribute on the operator (e.g. resumable: ClassVar[bool] = True), inference from BaseOperator subclass traits (e.g. presence of defer() / execute_complete()), or curated metadata in provider.yaml. A class attribute is the most discoverable and authoritative.
Registry surfacing. Badge on the operator detail page + a filter chip on the operator list page. Tooltip explaining what "resumable" means with a link to docs.
Implementation likely spans two repos:
apache/airflow -- metadata on the operators
apache/airflow-site -- rendering in landing-pages/ (registry)
Out of scope
Implementing resumability in operators that don't have it. This issue is about surfacing the capability, not adding it.
Problem
Users browsing the Airflow Registry cannot tell which operators support resumption -- i.e., operators that can pick up from where they left off after a worker crash, restart, or retry, rather than re-running from scratch.
This matters for users planning long-running or expensive workloads (data transfers, ML training, large queries) where re-running from scratch on retry is wasteful or unsafe.
Proposal
Add a visible indicator on operator pages (and a filter on listing pages) in the Airflow Registry that marks operators as resumable.
Open questions to resolve in this issue:
resumable: ClassVar[bool] = True), inference fromBaseOperatorsubclass traits (e.g. presence ofdefer()/execute_complete()), or curated metadata inprovider.yaml. A class attribute is the most discoverable and authoritative.Implementation likely spans two repos:
apache/airflow-- metadata on the operatorsapache/airflow-site-- rendering inlanding-pages/(registry)Out of scope
Project tracking
AIP-103: Task State Management