You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Issue with on_failure_callback and Downstream Retry in Airflow DAG
#46120
The issue is that when the dag fails, for example on task2, for the first try it captures task2 as the failed task. But for the second try, it captures one of the downstream tasks as the failed task.
Why does the on_failure_callback capture a downstream task as the failed task on the second retry?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi,
I have a dag with on_failure_callback function that writes the error to a BigQuery table.
The Airflow has 1 retry.
default_args = {
'owner': 'airflow',
'start_date': datetime(2023, 1, 1),
'retries': 1,
'retry_delay': timedelta(seconds=10),
'on_failure_callback': insert_failure_log_into_bq
}
with DAG(
dag_id='dag',
default_args=default_args,
schedule_interval=None,
catchup=False,
on_success_callback=None,
on_failure_callback=insert_failure_log_into_bq
) as dag:
def insert_failure_log_into_bq(context):
task_instance = context.get('task_instance')
dag_id = context.get('dag').dag_id
task_id = task_instance.task_id
....
task1 >> task2 >> ... Task10
The issue is that when the dag fails, for example on task2, for the first try it captures task2 as the failed task. But for the second try, it captures one of the downstream tasks as the failed task.
Why does the on_failure_callback capture a downstream task as the failed task on the second retry?
I use PythonOperator and BigQueryOperators.
All reactions