Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add 'queued' state to DagRun #16401

Merged
merged 4 commits into from Jul 6, 2021
Merged

Add 'queued' state to DagRun #16401

merged 4 commits into from Jul 6, 2021

Conversation

ephraimbuddy
Copy link
Contributor

Closes: #9975, #16366,
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added area:Scheduler Scheduler or dag parsing Issues area:webserver Webserver related Issues labels Jun 11, 2021
@ephraimbuddy ephraimbuddy mentioned this pull request Jun 11, 2021
Closed
@ephraimbuddy ephraimbuddy force-pushed the add-queued-state-for-dagrun branch 4 times, most recently from 92895fb to 5da31bb Compare June 14, 2021 13:44
@ephraimbuddy ephraimbuddy marked this pull request as ready for review June 14, 2021 13:48
@ephraimbuddy
Copy link
Contributor Author

dagrun-state

bbovenzi
bbovenzi approved these changes Jun 14, 2021
View reviewed changes
Copy link
Contributor

@bbovenzi bbovenzi left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just tested it locally. LGTM, from a UI/UX perspective.

@ephraimbuddy ephraimbuddy force-pushed the add-queued-state-for-dagrun branch 2 times, most recently from d358e0c to 32e63dd Compare June 14, 2021 21:58
ashb
ashb reviewed Jun 15, 2021
View reviewed changes
Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review, I want to look in more detail at the tests you've added.

airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/www/views.py Outdated Show resolved Hide resolved
tests/api_connexion/endpoints/test_dag_run_endpoint.py Show resolved Hide resolved
airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/models/dagrun.py Outdated Show resolved Hide resolved
@ephraimbuddy ephraimbuddy force-pushed the add-queued-state-for-dagrun branch 2 times, most recently from 3da1d46 to f336074 Compare June 15, 2021 17:21
@ephraimbuddy ephraimbuddy force-pushed the add-queued-state-for-dagrun branch 3 times, most recently from 0837146 to 2550638 Compare June 17, 2021 06:33
@kaxil kaxil added this to the Airflow 2.1.3 milestone Jul 6, 2021
@kaxil
Copy link
Member

kaxil commented Jul 6, 2021

Very well done @ephraimbuddy 👏

@fj-sanchez
Copy link
Contributor

Really nice to see this merged!! 🎉

So, is this planned for 2.1.3 instead of 2.1.2?

@ashb
Copy link
Member

ashb commented Jul 7, 2021

This will be 2.1.3 -- 2.1.2 is a quick follow on fix due to breaking Triggering DAGs via the UI, so we wan't to get that out quicker (i.e. 2.1.2 is a small change on 2.1.1, and this is a bigger change that needs a bit more testing time 😄 )

@uranusjr
Copy link
Member

uranusjr commented Jul 7, 2021

Yes. The timeline didn't change, but 2.1.2 was assigned to fix a critical issue in 2.1.1, so everything that was supposed to be in 2.1.2 automatically belongs to 2.1.3 now.

@ashb ashb mentioned this pull request Jul 7, 2021
Merged
jhtimmins pushed a commit that referenced this pull request Aug 11, 2021
@ephraimbuddy @jhtimmins
Add 'queued' state to DagRun (#16401)
f7bece3 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
kaxil pushed a commit that referenced this pull request Aug 13, 2021
@ephraimbuddy @kaxil
Add 'queued' state to DagRun (#16401)
2bab3d4 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
kaxil pushed a commit that referenced this pull request Aug 13, 2021
@ephraimbuddy @kaxil
Add 'queued' state to DagRun (#16401)
a682d1c 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
kaxil pushed a commit that referenced this pull request Aug 14, 2021
@ephraimbuddy @kaxil
Add 'queued' state to DagRun (#16401)
ee67daa 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
kaxil pushed a commit that referenced this pull request Aug 17, 2021
@ephraimbuddy @kaxil
Add 'queued' state to DagRun (#16401)
95ef3e3 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
jhtimmins pushed a commit that referenced this pull request Aug 17, 2021
@ephraimbuddy @jhtimmins
Add 'queued' state to DagRun (#16401)
9119b98 
This change adds queued state to DagRun. Newly created DagRuns
start in the queued state, are then moved to the running state after
satisfying the DAG's max_active_runs. If the Dag doesn't have
max_active_runs, the DagRuns are moved to running state immediately

Clearing a DagRun sets the state to queued state

Closes: #9975, #16366
(cherry picked from commit 6611ffd)
@kaxil kaxil mentioned this pull request Aug 18, 2021
Merged
kaxil added a commit to astronomer/airflow that referenced this pull request Aug 18, 2021
@kaxil
Sync DB Migrations with Airflow 2.1.3
810422b 
Since apache#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2
kaxil added a commit that referenced this pull request Aug 19, 2021
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
32475fa 
Since #16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2
@denysivanov denysivanov mentioned this pull request Sep 21, 2021
Closed
2 tasks
@eladkal eladkal mentioned this pull request Nov 22, 2021
Closed
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Mar 10, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
cac2339 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jun 4, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
e205fab 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jul 10, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
70965ae 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Aug 27, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
29631ab 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 4, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
46a1914 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
aglipska pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 7, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
b354bb2 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Dec 7, 2022
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
192a380 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jan 27, 2023
@kaxil
Sync DB Migrations with Airflow 2.1.3 (#17703)
de06878 
Since apache/airflow#16401 was backported to Airflow 2.1.3, the order of DB migrations needs to be changed in the `main` branch as the other PRs with DB migrations weren't ported and would be released in 2.2.

This PR sync the migrations to allow ugprade from 2.1.3 to 2.2

GitOrigin-RevId: 32475facce68a17d3e14d07762f63438e1527476
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@uranusjr uranusjr uranusjr left review comments

@ashb ashb ashb approved these changes

@bbovenzi bbovenzi bbovenzi approved these changes

@kaxil kaxil kaxil approved these changes

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@turbaszek turbaszek Awaiting requested review from turbaszek

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@mik-laj mik-laj Awaiting requested review from mik-laj

Assignees
No one assigned
Labels
area:Scheduler Scheduler or dag parsing Issues area:webserver Webserver related Issues full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Milestone
Airflow 2.1.3
Development

Successfully merging this pull request may close these issues.

max_active_runs = 1 can still create multiple active execution runs
6 participants
@ephraimbuddy @fj-sanchez @kaxil @ashb @uranusjr @bbovenzi