Support mark success for both upstream task and downstream mapped tasks in dag.test()#66983
Open
kevinhongzl wants to merge 3 commits into
Open
Conversation
c7bcc6b to
44eeec6
Compare
Contributor
SameerMesiah97
left a comment
There was a problem hiding this comment.
The fix looks plausible to me.
My only concern is around the map_index=0 fallback when upstream expansion metadata does not exist. It makessense pragmatically for dag.test(), but this touches mapped TI materialization internals. I’d defer to a maintainer for final approval.
44eeec6 to
cd795da
Compare
cd795da to
7e82cf6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Previously marking both an upstream task and its downstream mapped tasks as successful could still fail during expansion. Although the mapped task was marked successful, we still needed upstream XComs to determine how many task instances to create. When the upstream task was also marked successful and therefore did not run, no XCom was produced, so expansion failed. This PR updates
dag.test()to handle this scenario.closes: #61558
Updated behavior
The behavior is now:
In other words, when a mapped task is marked successful:
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.