[AIRFLOW-910] Use parallel task execution for backfills#2145
Closed
bolkedebruin wants to merge 1 commit intoapache:masterfrom
Closed
[AIRFLOW-910] Use parallel task execution for backfills#2145bolkedebruin wants to merge 1 commit intoapache:masterfrom
bolkedebruin wants to merge 1 commit intoapache:masterfrom
Conversation
The refactor to use dag runs in backfills caused a regression
in task execution performance as dag runs were executed
sequentially. Next to that, the backfills were non deterministic
due to the random execution of tasks, causing root tasks
being added to the non ready list too soon.
This updates the backfill logic as follows:
* Parallelize execution of tasks
* Use a leave first execution model; Breadth-first algorithm by Jerermiah
* Replace state updates from the executor by task based only
|
@bolkedebruin, thanks for your PR! By analyzing the history of the files in this pull request, we identified @mistercrunch, @jlowin and @plypaul to be potential reviewers. |
Codecov Report
@@ Coverage Diff @@
## master #2145 +/- ##
==========================================
+ Coverage 67.17% 67.19% +0.01%
==========================================
Files 142 142
Lines 10769 10769
==========================================
+ Hits 7234 7236 +2
+ Misses 3535 3533 -2
Continue to review full report at Codecov.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please accept this PR that addresses the following issues:
Testing Done:
specify why you think this is not required. We like to improve our
coverage so a non existing test is even a better reason to include one.
Reminders for contributors (REQUIRED!):
Your PR's title must reference an issue on
Airflow's JIRA.
For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA
issue Improving the search functionality in the graph view #1. Please open a new issue if required!
For all PRs with UI changes, you must provide screenshots. If the UI changes are not obvious, either annotate the images or provide before/after screenshots.
Please squash your commits when possible and follow the How to write a good git commit message.
Summarized as follows: