Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding ability to automatically set DAG to off after X times it failed sequentially #36935

Merged
merged 42 commits into from
Mar 15, 2024
Merged

Adding ability to automatically set DAG to off after X times it failed sequentially #36935

merged 42 commits into from
Mar 15, 2024

Conversation

pateash
Copy link
Contributor

@pateash pateash commented Jan 21, 2024 •
edited
Loading

closes: #36239

Description

This PR adds ability to automatically turn off DAG after X number of consecutive failures.

There are 2 ways to configure this

  1. in DAG() using max_consecutive_failed_dag_runs parameter this will override any config (as below)
  2. Using core config inside airflow.cfg or variable AIRFLOW__CORE_MAX_CONSECUTIVE_FAILED_DAG_RUNS_PER_DAG
    environment variable.

@eladkal eladkal changed the title 36239 automatically set dag off Add ability to automatically set DAG to off after X times it failed sequentially Jan 21, 2024
@eladkal eladkal added this to the Airflow 2.9.0 milestone Jan 21, 2024
@eladkal eladkal added the type:new-feature Changelog: New Features label Jan 21, 2024
dirrao
dirrao suggested changes Jan 24, 2024
View reviewed changes
Copy link
Collaborator

@dirrao dirrao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work. Can you add the test cases for this feature?

airflow/models/dagrun.py Outdated Show resolved Hide resolved
BasPH
BasPH reviewed Jan 24, 2024
View reviewed changes
Copy link
Contributor

@BasPH BasPH left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be a helpful feature. Left a few comments around naming conventions

airflow/config_templates/config.yml Outdated Show resolved Hide resolved
airflow/config_templates/config.yml Outdated Show resolved Hide resolved
airflow/models/dag.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
@pateash pateash changed the title Add ability to automatically set DAG to off after X times it failed sequentially Adding ability to automatically set DAG to off after X times it failed sequentially Jan 27, 2024
@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from d9d29c2 to 3ff75bf Compare February 1, 2024 10:42
@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from 8e54af0 to 6df778f Compare February 5, 2024 06:15
@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch from 394d028 to 6d7e91d Compare February 12, 2024 18:48
@pateash pateash marked this pull request as ready for review February 13, 2024 06:02
@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch from e836d92 to 081176f Compare February 13, 2024 06:03
@pateash pateash requested review from BasPH and dirrao February 13, 2024 06:03
@pateash
Copy link
Contributor Author

pateash commented Feb 13, 2024 •
edited
Loading

TODO:

  • adding test for core config
  • adding test for environment variable

@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from 80170e4 to 81ed11f Compare February 13, 2024 14:35
@pateash pateash requested a review from potiuk as a code owner February 13, 2024 18:34
@pateash pateash force-pushed the 36239-automatically-set-DAG-off branch 2 times, most recently from dfaa53a to f34cd9e Compare February 14, 2024 07:10
@eladkal eladkal force-pushed the 36239-automatically-set-DAG-off branch from a4af994 to bc56ee4 Compare March 15, 2024 10:51
@eladkal eladkal merged commit 777a216 into apache:main Mar 15, 2024
53 checks passed
@eladkal
Copy link
Contributor

eladkal commented Mar 15, 2024

🎉 🎉 🎉

@pateash pateash deleted the 36239-automatically-set-DAG-off branch March 15, 2024 12:58
@pateash
Copy link
Contributor Author

pateash commented Mar 15, 2024

🎉 🎉 🎉

Thanks a lot @eladkal

@o-nikolas
Copy link
Contributor

Looks like the diagrams weren't updated along with this migration? When you run precommit locally now, diagram changes are made for code that is unrelated.

@o-nikolas o-nikolas mentioned this pull request Mar 15, 2024
Merged
This was referenced Mar 17, 2024
Merged
Merged
@bbovenzi bbovenzi mentioned this pull request Mar 20, 2024
Open
2 tasks
utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Apr 22, 2024
@pateash @eladkal @utkarsharma2
Adding ability to automatically set DAG to off after X times it faile…
1d9015d 
…d sequentially (apache#36935)

---------

Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eladkal eladkal eladkal approved these changes

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@BasPH BasPH Awaiting requested review from BasPH

@dirrao dirrao Awaiting requested review from dirrao

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh

@uranusjr uranusjr Awaiting requested review from uranusjr

Assignees
No one assigned
Labels
area:API Airflow's REST/HTTP API area:CLI area:serialization type:new-feature Changelog: New Features
Projects
None yet
Milestone
Airflow 2.9.0
Development

Successfully merging this pull request may close these issues.

Add ability to automatically set DAG to off after X times it failed sequentially
8 participants
@pateash @potiuk @eladkal @o-nikolas @uranusjr @BasPH @amoghrajesh @dirrao