Resolve mapped task group arguments in task worker #27678
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This involves a few things: * The mapped task group needs to implement parse-time and run-time ti count accessors, similar to the mapped operator. * Since a non-mapped operator can be in a mapped task group and thus needs to be expanded, the parse-time and run-time ti count accessors are "raised" from MappedOperator to AbstractOperator (so they can be used by BaseOperator). These now also considers an operator's parent task groups. * If a ti count accessor is called on a non-mapped operator that's not in any mapped task group, the accessor should return a special value of a sort. Since None is already used, we raise an exception in this case instead (simply called NotMapped). For consistency, the previous case where None was returned (upstreams not available) is also changed to raise an exception instead (reusing NotFullyPopulated). This means the ti count accessors now always return an int, and raise an exception whenever the count is not available. * All of the above result in a coherant interface in AbstractOperator that can be used in DagRun to perform ti count revision. I also removed caching from get_mapped_ti_count since (judging from the various cache-clearing calls) it seems to be cause more trouble than benefits.
This is relatively simple in concept, mostly just involving moving things around and minor refactoring so ti expansion now works for both MappedOperator (already does) and BaseOperator in a MappedTaskGroup.
This reverts commit 8660be0.
This includes everything. I'll probably split this up after the scheduler part is merged since there's a lot going on here.
This was referenced Nov 15, 2022
|
Everything here is split out into separate PRs. (See references above.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #27491. There’s a lot going on here, I’ll split things up after that other PR is merged.