Description
In airflow backfill, -I is available so that the task instance of the first DAG run will ignore depends_on_past setting. It is a bit confusing from its name that it is a concept only for backfill jobs. For scheduled jobs, this behavior is on by default. However, one case we've been seeing a lot is that when a new task with depends_on_past=True was added, it will not be scheduled until its previous TI is marked as success. This is because when we are checking the taks deps, we consider a task instance the first task instance only when it's from the first DAG run of the DAG(roughly), code.
Use case / motivation
So that after adding new task, people don't need to remember manually marking the previous TI as success before getting the new task running.
Proposed behavior
Add a new property ignore_first_depends_on_past in BaseOperator and control its value through a config entry. If it is true, then first task instance of a task will not respect depends_on_past setting. A task instance will be considered as the first task instance of a task when there is not task instance in the DB with an execution_date earlier than it.
Or, making it the default behavior.
Related Issues
No
Description
In
airflow backfill,-Iis available so that the task instance of the first DAG run will ignore depends_on_past setting. It is a bit confusing from its name that it is a concept only for backfill jobs. For scheduled jobs, this behavior is on by default. However, one case we've been seeing a lot is that when a new task withdepends_on_past=Truewas added, it will not be scheduled until its previous TI is marked as success. This is because when we are checking the taks deps, we consider a task instance the first task instance only when it's from the first DAG run of the DAG(roughly), code.Use case / motivation
So that after adding new task, people don't need to remember manually marking the previous TI as success before getting the new task running.
Proposed behavior
Add a new property
ignore_first_depends_on_pastin BaseOperator and control its value through a config entry. If it is true, then first task instance of a task will not respectdepends_on_pastsetting. A task instance will be considered as the first task instance of a task when there is not task instance in the DB with an execution_date earlier than it.Or, making it the default behavior.
Related Issues
No