Trigger rule for "always"

[darwinyip]

Description

Trigger rule for always.

Use case / motivation

This comes in handy when there is a need to clean up resources at the end of the DAG run no matter whether it fails or succeeds.

For example:

allocate >> a >> b >> c >> d >> cleanup

The biggest issue is that if an error occurs before d, cleanup will never get executed.

Example use case is using BigQuery's Reservation API. We want to reserve slots at the beginning of the run and deallocate at the end. Not deallocating would mean a rampant bill at the end of the month.

[darwinyip]

Current workaround

with DAG(DAG_NAME, schedule_interval='@daily', default_args=default_args) as dag:
    task1 = DummyOperator(task_id='task1')
    task2 = BashOperator(task_id='task2', bash_command='exit 1')
    task3 = DummyOperator(task_id='task3')
​
    task1 >> task2 >> task3
​
    sensor = ExternalTaskSensor(task_id='sensor',
                                external_dag_id=DAG_NAME,
                                external_task_id='task3',
                                allowed_states=[State.SUCCESS, State.FAILED, State.UPSTREAM_FAILED],
                                mode='reschedule')
    last = DummyOperator(task_id='last')
​
    sensor >> last

[eladkal]

I have similar scenario:
create machine -> run steps -> terminate machine

For termination I use trigger rule all_done and it works great.
Whether upstream succeeded or failed the termination task is executed.
Won't it solve your use case as well?

[darwinyip]

That's true. Closing issue.