Skip to content

feat: implement DagTaskGroupsExistence and DagTasksExistence endpoints#67832

Open
dzpan0 wants to merge 2 commits into
apache:mainfrom
dzpan0:pr-64394-follow-up-deferrable-task-sensor
Open

feat: implement DagTaskGroupsExistence and DagTasksExistence endpoints#67832
dzpan0 wants to merge 2 commits into
apache:mainfrom
dzpan0:pr-64394-follow-up-deferrable-task-sensor

Conversation

@dzpan0
Copy link
Copy Markdown
Contributor

@dzpan0 dzpan0 commented Jun 1, 2026

Description

Follow-up to #64394, which fixed ExternalTaskSensor's check_existence=True for the AF2. On AF3, the deferrable path still defers silently when the external Dag, task, or task group does not exist, because the existence check has no access to the metadata database.

This PR adds two Execution API endpoints that let the sensor verify external references in a single round trip, and wires the sensor to use them before deferring:

  • GET /dags/{dag_id}/tasks/existence?task_ids=...
  • GET /dags/{dag_id}/task-groups/existence?task_group_ids=...

Each returns its input partitioned into existing and missing. Passing an empty task_ids list works as a Dag-existence probe (a missing Dag surfaces as DAG_NOT_FOUND, which the sensor translates to ExternalDagNotFoundError). Missing tasks or task groups surface as ExternalTaskNotFoundError and ExternalTaskGroupNotFoundError, respectively, matching the AF2 path's behavior.

The Task SDK exposes them via two new RuntimeTaskInstance accessors: get_dag_tasks_existence and get_dag_task_groups_existence.

Why

Existing dag-level Execution API routes can't distinguish a Dag that doesn't exist from a Dag that exists, but has no DagRuns: GET /dag_runs/count returns 0 and GET /dag_runs/previous returns None in both cases. GET /dags/{dag_id} settles the existence question on its own, but says nothing about specific tasks or task groups. These new endpoints fill that gap by combining the existence check with a per-id partition.

These endpoints can answer that cheaply and explicitly, which is useful for any future operator or sensor that needs to validate references against an external dag before deferring or scheduling work. The partitioned response shape (existing / missing) is also more informative than per-id checks when validating a list. Callers get one round trip instead of N, and a single response covers all four outcomes (all present, all missing, partial overlap, dag itself missing).

Testing

Unit tests added across the API route, datamodels, SDK client, supervisor handler, runtime accessor and sensor layers.

related: #40745


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    Claude (Opus 4.7) following the project guidelines.

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Adds two GET endpoints to the Execution API for batched existence
checks against a Dag's tasks and task groups:

Each takes a list of ids and returns them partitioned into 'existing'
and 'missing' with 200 or 404 only when the Dag is missing.
Passing an empty list works as a Dag existence probe.

These allow the clients to get the actual information of a Dag, returning
correct information even when the Dag hasn’t been ran once

related: apache#40745.

Co-authored-by: Diogo Callado <diogo.callado@tecnico.ulisboa.pt>
The new existence check and tests for ExternalTaskSensor uses endpoints
that were newly added, so these will be excluded for lower versions.
@dzpan0 dzpan0 marked this pull request as ready for review June 1, 2026 13:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant