Run a DAG from command line

[dirkgently007]

Is there a way to run a DAG from command line?

I see two options:

1. Use backfill and give same start and end date.
2. Use run - possible only if task_id is optional. Currently, run seems to support only running a specific task within a specific DAG. Is it possible to make task_id optional, so that if it's not provide, the run command will run the full DAG?

Is there any other way of doing this that I may have missed in the documentation?

[artwr]

I believe backfill is the canonical way, run being indeed reserved for a particular task in a dag. Note that to save a few keystrokes, you can only provide one of (start_date, end_date) to the backfill command and the other will be set to the same date.

[dirkgently007]

Thanks @artwr . That helps!

[mistercrunch]

The scheduler is the way to run your day to day processes, and backfill is the way to go if you need to backfill or reprocess parts of the DAG in time. Note that backfill can be used after a clear command to fill in the holes created by clear.

Say if you want to reprocess a specific task for a date range (that had succeeded but was some sort of false positive, or the business logic has changed) and all of the tasks downstream from this task, you probably want to start with a clear command that clears what needs to be reprocessed (here a set of tasks and everything downstream from it), and then run a backfill to fill in the holes in the right order.

Note that if you don't specify --end_date (-e) in the backfill command it will just run what you specified in start_date.

[dirkgently007]

Thanks @mistercrunch - that makes sense.