Skip to content

Surface resumable operators in the Airflow Registry #67236

@kaxil

Description

@kaxil

Problem

Users browsing the Airflow Registry cannot tell which operators support resumption -- i.e., operators that can pick up from where they left off after a worker crash, restart, or retry, rather than re-running from scratch.

This matters for users planning long-running or expensive workloads (data transfers, ML training, large queries) where re-running from scratch on retry is wasteful or unsafe.

Proposal

Add a visible indicator on operator pages (and a filter on listing pages) in the Airflow Registry that marks operators as resumable.

Open questions to resolve in this issue:

  • Definition. What qualifies as "resumable"? Candidates: deferrable operators (trigger-based resumption), operators with explicit checkpointing, idempotent transfer operators that can resume from offset/cursor, or some combination. Pick one definition or model it as multiple flags.
  • Source of truth. How do we get this signal? Options: a class attribute on the operator (e.g. resumable: ClassVar[bool] = True), inference from BaseOperator subclass traits (e.g. presence of defer() / execute_complete()), or curated metadata in provider.yaml. A class attribute is the most discoverable and authoritative.
  • Registry surfacing. Badge on the operator detail page + a filter chip on the operator list page. Tooltip explaining what "resumable" means with a link to docs.

Implementation likely spans two repos:

  • apache/airflow -- metadata on the operators
  • apache/airflow-site -- rendering in landing-pages/ (registry)

Out of scope

  • Implementing resumability in operators that don't have it. This issue is about surfacing the capability, not adding it.
  • Changing how deferrable operators work.

Project tracking

AIP-103: Task State Management

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

Status

Backlog

Status

Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions