AIP-115 Add on-demand task sections using skipped tasks

[dheerajturaga]

Summary

This draft implements the skipped-task alternative for AIP-115: On-Demand Task Sections.

It is intentionally opened alongside #71038 so the PMC can compare two lifecycle models for the same user-facing feature:

• AIP-115 Add on-demand task sections #71038 gives intentionally omitted work a dedicated bypassed state.
• This draft represents intentionally omitted work with Airflow's existing skipped state.

Both proposals let Dag authors keep expensive, risky, slow, or rarely needed work visible in a Dag without running it automatically. A normal Dag run can finish without waiting for that optional work, while an authorized user can later run the section for the exact Dag run that needs it.

Motivation

Airflow supports related workflows today, but none directly express non-blocking work that remains available on demand:

• Human-in-the-loop tasks deliberately keep a Dag run open while awaiting a response. On-demand sections do not block the run.
• Branching and short-circuiting can omit work based on runtime logic, but do not communicate that users may run the omitted section later.
• Clearing task instances can rerun part of a Dag, but requires users to understand the graph and select the correct task instances and options themselves.

On-demand sections make this intent explicit in the Dag definition and provide a discoverable, scoped action in the UI and API.

Authoring experience

Dag authors place an OnDemandSectionOperator at the start of an optional path and may give the section a user-facing label:

from airflow.providers.standard.operators.on_demand import OnDemandSectionOperator

production_release = OnDemandSectionOperator(
    task_id="production_release",
    label="Deploy this release to production",
)

verify_staging >> production_release >> deploy_to_production >> smoke_test

The label identifies the action to users. When omitted, the task display name or task_id is used.

By default, the section controls all downstream descendants. Authors may instead limit it to direct downstream tasks, allowing later descendants to follow their existing trigger rules. Teardown tasks remain outside the optional section so cleanup behavior is preserved.

Dag run behavior

For scheduled, manually triggered, and backfill Dag runs:

1. The required path runs normally.
2. The on-demand section marker succeeds.
3. Tasks controlled by the section are skipped by default.
4. The Dag run can reach a terminal state without waiting for user input or optional work.

This means backfills and manual runs do not unexpectedly execute expensive or high-impact optional sections. The section remains visible in the graph and retains a clear action for the specific Dag run.

Running a section

After the section marker succeeds, users can run its section from Graph view or the task instance details for that selected Dag run. The action never implicitly targets the latest run.

Before confirmation, Airflow previews the affected task instances. The confirmation flow supports an action note and protects already-running task instances by default. After confirmation:

• the selected section becomes eligible for normal scheduling;
• the Dag run resumes if it had already completed;
• the section marker remains successful;
• downstream dependencies and trigger rules continue to apply; and
• tasks retain their normal retries, logs, callbacks, pools, queues, executor behavior, XComs, and observability.

A dedicated public task-instance action provides both preview and execution modes and returns the affected task instances. Invalid targets and unsafe transitions produce explicit client errors rather than silently acting on unrelated work.

The action is available for successful, non-mapped on-demand section task instances. A section can be run again when needed, subject to the default protection against disrupting active tasks.

Permissions and auditability

Running an on-demand section is a mutating task-instance action. It requires the existing Dag task-instance mutation permission and participates in normal API action logging.

This feature does not add named approvers, assignees, or task-level authorization. Deployments that need a person or group to approve execution should continue to use Human-in-the-loop operators.

CI/CD example

The included example models a release pipeline that builds and tests an artifact, publishes it, deploys it to staging, and verifies staging during the normal Dag run. Production deployment and production smoke tests remain visible but skipped.

After reviewing staging, an authorized operator can run the production section for that exact release's Dag run. This avoids rebuilding the artifact, keeps the production path attached to its originating release, and prevents backfills from deploying historical releases automatically.

Other suitable workflows include optional enrichment, report generation, historical reconciliation, deep validation, one-off exports, and high-impact follow-up work that should require an explicit operational action.

Design tradeoff for PMC review

Using skipped keeps the task-state model smaller and reuses behavior already understood by the scheduler, UI, APIs, executors, and users. It also avoids adding a state that every task-state consumer must recognize.

The tradeoff is that a generic task-state view cannot distinguish an on-demand omission from other skipped work using state alone. The Dag definition, operator identity, section label, and dedicated action provide that context, but analytics based only on task state will see skipped.

The central decision between this draft and #71038 is therefore whether first-class state-level distinction is valuable enough to justify expanding the task-state model. The authoring and operational experience can otherwise remain substantially the same.

Compatibility and scope

This proposal is additive. Existing Dags, skipped-task behavior, branching, Human-in-the-loop workflows, and executor contracts remain unchanged unless a Dag adopts OnDemandSectionOperator.

It intentionally does not:

• make arbitrary skipped tasks manually runnable;
• replace blocking approval workflows;
• introduce a separate Dag run type;
• automatically run optional sections in backfills or manual Dag runs; or
• add task-level authorization beyond existing Dag and task-instance permissions.

Validation

• 11 focused provider and public API tests pass.
• 19 focused Graph view and task instance UI tests pass.
• OpenAPI, Airflow CLI datamodel, permissions, formatting, lint, and repository consistency checks pass.

related: #71038

---

Was generative AI tooling used to co-author this PR?

• Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

---

Drafted-by: Codex (GPT-5) (no human review before posting)