Fix Scheduler crash looping when dagrun creation fails #35135
Merged
+44
−11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ephraimbuddy
force-pushed
the
fix-scheduler-crash-on-dagrun-creation
branch
from
d253f43
to
42190ac
Compare
Scheduler was found crash looping with the error: airflow.exceptions.AirflowException: Creating DagRun needs either "run_id™ or both "run_type and "execution_date'. This happened in an old airflow(2.3.4) and I can't reproduce it on the latest but exception from dag.create_dagrun should not crash the scheduler. Instead, the scheduler should catch the exception and log it. This PR provides that behaviour
ephraimbuddy
force-pushed
the
fix-scheduler-crash-on-dagrun-creation
branch
from
42190ac
to
c82043c
Compare
phanikumv
approved these changes
Oct 23, 2023
jedcunningham
approved these changes
Oct 23, 2023
eladkal
approved these changes
Oct 23, 2023
potiuk
approved these changes
Oct 23, 2023
ephraimbuddy
added a commit
that referenced
this pull request
Oct 29, 2023
(cherry picked from commit 98618f0)
ephraimbuddy
added a commit
that referenced
this pull request
Oct 30, 2023
(cherry picked from commit 98618f0)
40 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Scheduler was found crash looping with the error:
airflow.exceptions.AirflowException: Creating DagRun needs either "run_id™ or both "run_type and "execution_date'.
This happened in an old airflow(2.3.4) and I can't reproduce it on the latest but exception from dag.create_dagrun should not crash the scheduler.
Instead, the scheduler should catch the exception and log it.
This PR provides that behaviour