Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CLI support for bulk pause and resume of DAGs #38265

Merged
merged 4 commits into from Mar 20, 2024

Conversation

shahar1
Copy link
Contributor

@shahar1 shahar1 commented Mar 18, 2024

closes: #36956

This PR adds the ability for airflow dags (un)pause commands to treat dag_id as a regex (similar to backfill), so we could apply it to multiple DAGs.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@shahar1
Copy link
Contributor Author

shahar1 commented Mar 18, 2024

Following @eladkal's comment in #36994, I added a confirmation prompt to avoid accidental (un)pausing. I also formatted the output with AirflowConsole().print_as to make it easier to read.

@shahar1 shahar1 marked this pull request as draft March 18, 2024 20:20
@shahar1 shahar1 marked this pull request as ready for review March 18, 2024 21:32
@shahar1 shahar1 force-pushed the pause-multiple-dags branch 3 times, most recently from 509bf86 to afa2bb6 Compare March 19, 2024 06:27
airflow/cli/cli_config.py Outdated Show resolved Hide resolved
airflow/cli/cli_config.py Outdated Show resolved Hide resolved
@shahar1 shahar1 requested a review from vincbeck March 20, 2024 09:27
@vincbeck vincbeck merged commit fecc1ed into apache:main Mar 20, 2024
45 checks passed
@eladkal eladkal added this to the Airflow 2.9.0 milestone Mar 22, 2024
@eladkal eladkal added the type:improvement Changelog: Improvements label Mar 22, 2024
Copy link
Contributor

@ephraimbuddy ephraimbuddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken

Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

@shahar1
Copy link
Contributor Author

shahar1 commented Mar 28, 2024

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken

Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

Apologies for the confusion, maybe raising an error for pausing an already paused dag (and vice versa) is indeed too aggressive - I'll try to come up with a fix later on.

@ephraimbuddy
Copy link
Contributor

It seems like unpausing a single dag is now broken by this PR. I have not tested bulk pause/unpause but for one dag, it seems broken
Ok. It now errors when the dag is paused and you are pausing again. I think it should just exit without error if DAG is already paused

Apologies for the confusion, maybe raising an error for pausing an already paused dag (and vice versa) is indeed too aggressive - I'll try to come up with a fix later on.

Added the change here: #38585. Take a look when you have time

utkarsharma2 pushed a commit to astronomer/airflow that referenced this pull request Apr 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI type:improvement Changelog: Improvements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add CLI support for bulk pause and resume of DAGs
5 participants