Add run_type to DagRun #8227

turbaszek · 2020-04-09T10:10:23Z

https://issues.apache.org/jira/browse/AIRFLOW-6242

Make sure to mark the boxes below before creating PR: [x]

Description above provides context of the change
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

airflow/models/dagrun.py

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py

airflow/api/common/experimental/mark_tasks.py

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py

ashb

dag.create_run() is a quasi-public API so we should probably mention this change in updating.md?

Looks good, just need to work out what is the best index to put on the new run_type column.

ashb · 2020-04-09T10:53:21Z

Oh and we should add the run_type column to the DAGRunModelView.

codecov-io · 2020-04-17T10:47:22Z

Codecov Report

Merging #8227 into master will increase coverage by 0.01%.
The diff coverage is 14.63%.

@@            Coverage Diff            @@
##           master   #8227      +/-   ##
=========================================
+ Coverage    6.22%   6.24%   +0.01%     
=========================================
  Files         946     941       -5     
  Lines       45695   45667      -28     
=========================================
+ Hits         2846    2851       +5     
+ Misses      42849   42816      -33

Impacted Files	Coverage Δ
airflow/api/common/experimental/mark_tasks.py	`0.00% <ø> (ø)`
airflow/api/common/experimental/trigger_dag.py	`0.00% <0.00%> (ø)`
airflow/jobs/backfill_job.py	`0.00% <0.00%> (ø)`
airflow/jobs/base_job.py	`0.00% <ø> (ø)`
airflow/jobs/scheduler_job.py	`0.00% <0.00%> (ø)`
airflow/operators/subdag_operator.py	`0.00% <0.00%> (ø)`
airflow/ti_deps/deps/dagrun_id_dep.py	`69.23% <0.00%> (-2.20%)`	⬇️
airflow/www/views.py	`0.00% <0.00%> (ø)`
airflow/models/dag.py	`26.47% <11.11%> (-0.18%)`	⬇️
airflow/models/dagrun.py	`27.19% <33.33%> (+0.58%)`	⬆️
... and 23 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9eaacbd...ed5618c. Read the comment docs.

ashb · 2020-06-01T11:05:34Z

airflow/api/common/experimental/trigger_dag.py

-            dag_id
-        ))
+        run_type = DagRunType.MANUAL
+        dag_run = dag_run.find(dag_id=dag_id, run_type=run_type, execution_date=execution_date)


Is there is already a scheduled dag_run for this exact time but of a different run_type, this find() will fail to find anything, but the unique constraint (on exeuction_date, dag_id) would still be violated.

I will fix it

airflow/models/dag.py

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py

ashb · 2020-06-01T11:12:08Z

@turbaszek Is it much work to run your benchmark again: DR.run_type != DagRunType.BACKFILL_JOB.value is something we do in the "hot" path of the scheduler, and the != isn't covered in your current benchmarks.

(It's hard for me to see the detail in the google sheet as in view only mode it won't even let me resize the columns. Thanks Google.)

Seriously? I've added edit access

Yeah, it's cos sometimes that makes sense cos all users share the same view (sorting, filtering, maybe you want to hide colums from some users etc). Annoying for like 80% of the time though!

turbaszek · 2020-06-02T14:26:27Z

@turbaszek Is it much work to run your benchmark again: DR.run_type != DagRunType.BACKFILL_JOB.value is something we do in the "hot" path of the scheduler, and the != isn't covered in your current benchmarks.

I will run those tests today / tomorrow

ashb · 2020-06-02T20:53:07Z

@turbaszek Is it much work to run your benchmark again: DR.run_type != DagRunType.BACKFILL_JOB.value is something we do in the "hot" path of the scheduler, and the != isn't covered in your current benchmarks.

I will run those tests today / tomorrow

Cool, thanks. I don't expect this to make a difference, but if it's easy to check let's do it

turbaszek · 2020-06-03T11:51:49Z

Assuming that we query all combination of (dag_id, run_id, state) + run_typ != Backfill we get

how many times given idex yielded the minimum time

 "Index('dag_id_state_type', dag_id, _state, run_type),": 0,
 "Index('dag_id_state', dag_id, _state),Index('dag_id_type', dag_id, run_type),": 3,
 "Index('dag_id_type', dag_id, run_type),": 2,
 "Index('dag_id_state', dag_id, _state),": 2

@ashb I've updated the spreadsheet, as expected there's nothing that would favor another index than the current one.

turbaszek · 2020-06-03T12:10:13Z

@kaxil @ashb I also addressed all your suggestions

ashb

One small change, then LGTM!

(Oh plus fixing the failing test 😉)

airflow/www/views.py

airflow/jobs/scheduler_job.py

turbaszek · 2020-06-03T17:42:18Z

Thanks @ashb , your comments are addressed 👌

fixup! Add run_type to DagRun fixup! fixup! Add run_type to DagRun fixup! fixup! fixup! Add run_type to DagRun fixup! fixup! fixup! Add run_type to DagRun fixup! Add run_type to DagRun fixup! Add run_type to DagRun Adjust TriggerDagRunOperator fixup! Adjust TriggerDagRunOperator Add index Make run_type not nullable Add type check for run_type fixup! Add type check for run_type

turbaszek · 2020-06-04T13:07:15Z

@ashb @kaxil @mik-laj I've fixed the migration for MySQL and SQLite

turbaszek · 2020-06-04T14:20:35Z

Thanks all!

potiuk · 2020-06-04T15:22:04Z

🎉

boring-cyborg bot added area:CLI area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues labels Apr 9, 2020

turbaszek commented Apr 9, 2020

View reviewed changes

airflow/models/dagrun.py Show resolved Hide resolved

turbaszek requested review from ashb and mik-laj April 9, 2020 10:15

turbaszek added area:MetaDB Meta Database related issues. area:performance and removed area:CLI area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues labels Apr 9, 2020

boring-cyborg bot added area:CLI area:Scheduler including HA (high availability) scheduler area:webserver Webserver related Issues labels Apr 9, 2020

mik-laj reviewed Apr 9, 2020

View reviewed changes

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py Outdated Show resolved Hide resolved

ashb reviewed Apr 9, 2020

View reviewed changes

airflow/api/common/experimental/mark_tasks.py Show resolved Hide resolved

ashb reviewed Apr 9, 2020

View reviewed changes

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py Show resolved Hide resolved

ashb reviewed Apr 9, 2020

View reviewed changes

turbaszek force-pushed the add-run-type-to-dag-run branch 6 times, most recently from 6e2e65e to 01b339e Compare April 15, 2020 16:25

turbaszek force-pushed the add-run-type-to-dag-run branch from 01b339e to a5b36b7 Compare April 17, 2020 10:03

turbaszek force-pushed the add-run-type-to-dag-run branch from a5b36b7 to ed5618c Compare April 23, 2020 12:21

turbaszek requested review from mik-laj and ashb April 23, 2020 12:23

ashb reviewed Jun 1, 2020

View reviewed changes

ashb requested changes Jun 1, 2020

View reviewed changes

airflow/models/dag.py Outdated Show resolved Hide resolved

airflow/migrations/versions/3c20cacc0044_add_dagrun_run_type.py Show resolved Hide resolved

turbaszek force-pushed the add-run-type-to-dag-run branch 2 times, most recently from 21e288c to 3a018fd Compare June 3, 2020 12:08

turbaszek force-pushed the add-run-type-to-dag-run branch from 3a018fd to 6bcf2ae Compare June 3, 2020 12:14

ashb requested changes Jun 3, 2020

View reviewed changes

airflow/www/views.py Outdated Show resolved Hide resolved

ashb reviewed Jun 3, 2020

View reviewed changes

airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved

turbaszek requested a review from ashb June 3, 2020 17:41

turbaszek force-pushed the add-run-type-to-dag-run branch from 34d27bc to 2a22ee0 Compare June 3, 2020 19:55

turbaszek requested a review from kaxil June 3, 2020 19:56

kaxil approved these changes Jun 3, 2020

View reviewed changes

turbaszek added 5 commits June 4, 2020 13:41

fixup! Add run_type to DagRun

eb1928e

fixup! fixup! Add run_type to DagRun

a89aabe

Fix migration

e28465f

fixup! Fix migration

70e9dd0

turbaszek force-pushed the add-run-type-to-dag-run branch from 6a5ff8c to 70e9dd0 Compare June 4, 2020 13:00

ashb approved these changes Jun 4, 2020

View reviewed changes

turbaszek merged commit 533b143 into apache:master Jun 4, 2020

turbaszek deleted the add-run-type-to-dag-run branch June 4, 2020 14:20

turbaszek mentioned this pull request Oct 6, 2020

Add job_id to DagRun table and remove BackfillJob zombies #11302

Open

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add run_type to DagRun #8227

Add run_type to DagRun #8227

turbaszek commented Apr 9, 2020 •

edited

Loading

ashb left a comment

ashb commented Apr 9, 2020

codecov-io commented Apr 17, 2020 •

edited

Loading

ashb Jun 1, 2020 •

edited

Loading

turbaszek Jun 2, 2020

ashb commented Jun 1, 2020 •

edited

Loading

turbaszek commented Jun 2, 2020

ashb commented Jun 2, 2020

turbaszek commented Jun 3, 2020

turbaszek commented Jun 3, 2020

ashb left a comment

turbaszek commented Jun 3, 2020

turbaszek commented Jun 4, 2020

turbaszek commented Jun 4, 2020

potiuk commented Jun 4, 2020

Add run_type to DagRun #8227

Add run_type to DagRun #8227

Conversation

turbaszek commented Apr 9, 2020 • edited Loading

ashb left a comment

Choose a reason for hiding this comment

ashb commented Apr 9, 2020

codecov-io commented Apr 17, 2020 • edited Loading

Codecov Report

ashb Jun 1, 2020 • edited Loading

Choose a reason for hiding this comment

turbaszek Jun 2, 2020

Choose a reason for hiding this comment

ashb commented Jun 1, 2020 • edited Loading

turbaszek commented Jun 2, 2020

ashb commented Jun 2, 2020

turbaszek commented Jun 3, 2020

turbaszek commented Jun 3, 2020

ashb left a comment

Choose a reason for hiding this comment

turbaszek commented Jun 3, 2020

turbaszek commented Jun 4, 2020

turbaszek commented Jun 4, 2020

potiuk commented Jun 4, 2020

turbaszek commented Apr 9, 2020 •

edited

Loading

codecov-io commented Apr 17, 2020 •

edited

Loading

ashb Jun 1, 2020 •

edited

Loading

ashb commented Jun 1, 2020 •

edited

Loading