-
Notifications
You must be signed in to change notification settings - Fork 13.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIP-39: Handle DAG scheduling with timetables #15397
Conversation
@uranusjr Can you explain why the |
I’ve actually removed |
c5a687a
to
ccf50ff
Compare
Alright, I’ve pushed my implementation to all the possible I’ve noticed a difficulty during my initial investigation though. There are various places using The simple solution would be to introduce something like |
Is your PR description (or the bit about "should not be merged") still accurate? |
No it’s not, thanks for catching this. I’ve edited the top message to reflect the current status. |
d97ff83
to
8486c10
Compare
Grr, not sure what’s going on the the Static Checks job, it just says “This check failed” for me and nothing else. The Postgres check failed with
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
b8301f9
to
e3916e3
Compare
2a7ddfd
to
c65e28b
Compare
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
05cb9fc
to
b46dace
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👏 Nice work 🥳 🎉
"""Get information about the next DagRun of this dag after ``date_last_automated_dagrun``. | ||
|
||
This calculates what time interval the next DagRun should operate on | ||
(its execution date), and when it can be scheduled, , according to the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(its execution date), and when it can be scheduled, , according to the | |
(its execution date), and when it can be scheduled, according to the |
e13fc8a
to
113dca0
Compare
This creates a new subpackage airflow.timetables, and implements timetable constructs that provides DAG scheduling logic. The timetable classes are used to refactor schedule inference logic out of the DAG class, and existing functions related to scheduling are refactored to use timetables (and deprecated). Usages of the deprecated DAG functions in Airflow's code base are modified to either use the timetable, or infer the information by other means. For example, usages of previous_schedule() (what was a DAG last scheduled to run before this run?) are refactored to query the database when the previous scheduled run actually happened, instead of using the schedule interval (cron or timedelta) in infer the information. This is because an AIP-39 timetable does not necessarily run on a periodic-ish schedule, and we cannot reliably infer when the previous run happened.
113dca0
to
264e3ee
Compare
I’m going to pull the trigger on this soon-ish if the checks pass; this one modifies way too many files and is causing merge conflicts left and right. |
f68e4ec
to
b6c95cb
Compare
Man it’s so difficult to see all checks pass together. I think I’ve seen each of them passed at least once during my multiple attempts though, so this is going in. |
0ebad6d
to
f5b55ee
Compare
🎉 |
Woohoo! |
🎉 |
This creates a skeleton for the AIP-39 implementation to build on, and refactor the logic handling
@once
schedules out of the DAG class.The current implementation contains a lot of duplicated code, and the PR probably shouldn’t be merged until I can pick most ofnext_dagrun_after_date
apart (and ultimately eliminate it entirely and rewrite all the tests againstDAG.next_dagrun_info()
or the various TimeTable classes instead).Update: I’ve since did that, so this PR is clean now.
I want to post this WIP since the trivial
@once
implementation already exposes things missing from the interface outlined in AIP-39, and I had to invent an argument (between
) to cover it. I want to know how ya’ll think of it 🙂(Note: The current
TimeTable.next_dagrun_info()
interface is missing the session argument from AIP-39 because I don’t need it yet. It will be added when it needs to be.)^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.