Skip to content

Support mark success for both upstream task and downstream mapped tasks in dag.test()#66983

Open
kevinhongzl wants to merge 3 commits into
apache:mainfrom
kevinhongzl:support-mark-success-for-mapped-tasks-in-dag-test
Open

Support mark success for both upstream task and downstream mapped tasks in dag.test()#66983
kevinhongzl wants to merge 3 commits into
apache:mainfrom
kevinhongzl:support-mark-success-for-mapped-tasks-in-dag-test

Conversation

@kevinhongzl
Copy link
Copy Markdown
Contributor

Summary

Previously marking both an upstream task and its downstream mapped tasks as successful could still fail during expansion. Although the mapped task was marked successful, we still needed upstream XComs to determine how many task instances to create. When the upstream task was also marked successful and therefore did not run, no XCom was produced, so expansion failed. This PR updates dag.test() to handle this scenario.

closes: #61558

Updated behavior

The behavior is now:

upstream (marked success) -> no XCom  -> mapped (marked success) -> create 1 successful TI
upstream (executed)       -> XCom [n] -> mapped (marked success) -> create n successful TIs
upstream (marked success) -> no XCom  -> mapped (executed) -> fail as before
upstream (executed)       -> XCom [n] -> mapped (executed) -> create n executed TIs

In other words, when a mapped task is marked successful:

  • if upstream mapping data exists, we use it to create the expected number of mapped task instances
  • if upstream mapping data does not exist because the upstream task was also marked successful, we create a single successful mapped task

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@choo121600 choo121600 added the ready for maintainer review Set after triaging when all criteria pass. label May 15, 2026
@kevinhongzl kevinhongzl force-pushed the support-mark-success-for-mapped-tasks-in-dag-test branch from c7bcc6b to 44eeec6 Compare May 15, 2026 18:08
Copy link
Copy Markdown
Contributor

@SameerMesiah97 SameerMesiah97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks plausible to me.

My only concern is around the map_index=0 fallback when upstream expansion metadata does not exist. It makessense pragmatically for dag.test(), but this touches mapped TI materialization internals. I’d defer to a maintainer for final approval.

@kevinhongzl kevinhongzl force-pushed the support-mark-success-for-mapped-tasks-in-dag-test branch from 44eeec6 to cd795da Compare May 21, 2026 02:51
@kevinhongzl kevinhongzl force-pushed the support-mark-success-for-mapped-tasks-in-dag-test branch from cd795da to 7e82cf6 Compare May 21, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:task-sdk ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dag.test "mark_success_pattern" could work nicer with dynamically mapped tasks

3 participants