Skip to content

Conversation

@SameerMesiah97
Copy link
Contributor

@SameerMesiah97 SameerMesiah97 commented Dec 17, 2025

Description

Ensure that tasks within a MappedTaskGroup which are implicitly dependent on mapping sources are added to the downstream depdencies of the mapping source, even when they are not explicitly defined as consumers of the mapping input.

This change ensures implicit dependencies of the mapping source within a MappedTaskGroup are consistently reflected at the task level.

Rationale

Previously, only tasks within a MappedTaskGroup that explicitly consumed the mapping input were registered as downstream dependencies of the mapping source.

For example, in the task group definition below, the mapped task created by calling the mapping function explicitly consumes the mapping function passed into the argument y:

task_group_task = process_something(x, y)

As a result, this task was correctly registered as downstream of the mapping source.

However, this was not the case for other tasks within the same MappedTaskGroup (but not immediately downstream of the mapping source) that did not directly consume the mapping input:

task_group_end = EmptyOperator (task_id ='end')

These relied solely on task-to-task relationships within the group:

'task_group_task >> task_group_end `

Even though the entire task group was downstream of the mapping source:

dag_level_start >> the_list >> mapped_group >> dag_level_end

Tasks such task_group_end did not inherit an implicit task-level dependency on the mapping source. Their dependency on the mapping input was only represented at the group level.

This could cause implicit dependencies to be skipped during DAG evaluation under certain trigger rules (such as NONE_FAILED_MIN_ONE_SUCCESS), particularly when multiple parallel task streams within the MappedTaskGroup converge on a single downstream task outside the group such as dag_level_end

As a result, DAG runs could be incorrectly marked as failed or skipped despite valid upstream execution.

Please refer to issue #59289 for more context. This PR was opened in response to that. The author of the issue reported the bug in Airflow 3.0.6 but I can confirm that the same issue is present in Airflow 3.1.5 after reproducing the bug.

Tests

Added a test covering MappedTaskGroup behavior to ensure all tasks within the group are registered as downstream dependencies of the mapping source, including tasks that do not explicitly consume the mapping input.

Closes: #59289

@SameerMesiah97 SameerMesiah97 changed the title Fix missing implicit downstream dependencies from mapping sources in TaskGroups Fix missing implicit downstream dependencies for mapping sources in MappedTaskGroups Dec 17, 2025
@SameerMesiah97 SameerMesiah97 marked this pull request as draft December 17, 2025 16:33
@SameerMesiah97
Copy link
Contributor Author

I closed this PR because it was violating an invariant (as evidenced by the failing tests) by forcing mapped tasks to depend on mapping functions they do not consume. This required complete rework so another PR (#59289) was opened with a different approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic Task Mapping Skips Before Upstream Has Started

1 participant