Skip to content

Fix per-index evaluation of ONE_FAILED in mapped task groups#67684

Merged
kaxil merged 2 commits into
apache:mainfrom
shahar1:fix-50210-trigger-rule-mapped-task-group
May 29, 2026
Merged

Fix per-index evaluation of ONE_FAILED in mapped task groups#67684
kaxil merged 2 commits into
apache:mainfrom
shahar1:fix-50210-trigger-rule-mapped-task-group

Conversation

@shahar1
Copy link
Copy Markdown
Contributor

@shahar1 shahar1 commented May 29, 2026

closes: #50210

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS, ONE_DONE) inside a mapped task group was evaluated against every expanded instance of its upstream, instead of the upstream instance that shares its own map index. As a result, a single failed (or succeeded) upstream instance wrongly triggered the rule for every expanded instance of the task.

For example, in the reproduction from the issue, divide(0) fails while divide(1..3) succeed, yet the mapped report_failure task (ONE_FAILED) ran for all map indexes instead of only index 0.

Root cause

TriggerRuleDep._get_relevant_upstream_map_indexes short-circuits to "depend on every upstream instance" for the fast-triggered rules when the upstream is not an expansion dependency of the group (introduced in #44937 to fix #34023). That broad behavior is only needed for the not-yet-expanded summary task instance (map_index < 0), so the task is not prematurely skipped before the mapped task group expands. Applying it to the already-expanded instances broke per-index correspondence.

Fix

Restrict the special case to the summary instance (ti.map_index < 0). Expanded instances now use the normal per-map-index upstream resolution, so each report_failure[k] depends on divide[k] only.

Tests


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in apache#34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.

closes: apache#50210
@shahar1 shahar1 marked this pull request as draft May 29, 2026 07:10
@shahar1 shahar1 added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label May 29, 2026
@shahar1 shahar1 marked this pull request as ready for review May 29, 2026 07:30
@shahar1
Copy link
Copy Markdown
Contributor Author

shahar1 commented May 29, 2026

Tested against the original repro. example in the issue (compiled into Airflow 3), seems to do the job:

from __future__ import annotations

import logging
from datetime import datetime

from airflow.sdk import DAG, task, task_group
from airflow.utils.trigger_rule import TriggerRule

logger = logging.getLogger(__name__)


@task
def divide(i):
    return 30 / i


@task(trigger_rule=TriggerRule.ONE_FAILED)
def report_failure(i):
    logger.info("%s failed", i)


@task
def report_success(i):
    logger.info("%s succeeded", i)


with DAG(
    dag_id="Test-Trigger-Rule",
    start_date=datetime(2024, 1, 1),
    schedule=None,
    catchup=False,
    tags=["test", "trigger-rule"],
) as dag:

    @task
    def gen_examples():
        return [0, 1, 2, 3]

    @task_group
    def divide_and_report(i):
        divide(i) >> [report_success(i), report_failure(i)]

    divide_and_report.expand(i=gen_examples())
image

Copy link
Copy Markdown
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@kaxil kaxil merged commit 862b647 into apache:main May 29, 2026
125 of 147 checks passed
@github-actions github-actions Bot added this to the Airflow 3.2.3 milestone May 29, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.2.3 based on: backport label targeting v3-2-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

@github-actions
Copy link
Copy Markdown
Contributor

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-2-test PR Link

@shahar1 shahar1 deleted the fix-50210-trigger-rule-mapped-task-group branch May 29, 2026 10:26
github-actions Bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 29, 2026
…ups (apache#67684)

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in apache#34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.
(cherry picked from commit 862b647)

Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
closes: apache#50210
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request May 29, 2026
…ups (apache#67684)

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in apache#34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.
(cherry picked from commit 862b647)

Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
closes: apache#50210
vatsrahul1001 pushed a commit that referenced this pull request May 29, 2026
…ups (#67684) (#67704)

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in #34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.
(cherry picked from commit 862b647)


closes: #50210

Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
vatsrahul1001 pushed a commit that referenced this pull request May 29, 2026
…ups (#67684) (#67704)

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in #34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.
(cherry picked from commit 862b647)


closes: #50210

Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
sunildataengineer pushed a commit to sunildataengineer/airflow that referenced this pull request May 30, 2026
…67684)

A task using a "fast triggered" trigger rule (ONE_FAILED, ONE_SUCCESS,
ONE_DONE) inside a mapped task group was evaluated against every expanded
instance of its upstream, instead of the upstream instance sharing its own
map index. As a result a single failed (or succeeded) upstream instance
wrongly triggered the rule for every expanded instance of the task — e.g. a
mapped ONE_FAILED reporting task ran for all map indexes when only one
upstream had failed.

The broad "depend on every upstream instance" behavior is only needed for
the not-yet-expanded summary task instance (map_index < 0), so a fast
trigger rule does not prematurely skip the task before the mapped task group
expands (the case fixed in apache#34023). Restrict that special case to the
summary instance; expanded instances now use the normal per-map-index
upstream resolution.

closes: apache#50210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected behavior of dynamically generated task with TriggerRule ONE_FAILED Trigger Rule ONE_FAILED does not work in task group with mapped tasks

2 participants