DebugExecutor runs task callbacks twice

[jochenott]

Apache Airflow version

2.3.3 (latest released)

What happened

The DebugExecutor runs the on_failure_callback twice.

This is due to TaskInstance._run_finished_callback() being called twice: once implicitly via the (new) call to ti.run(), once explicitly again:

https://github.com/apache/airflow/blob/62778bb8648515e9823ace2f48d9a89ec91a6803/airflow/executors/debug_executor.py#L79,L81

This came in to the airflow 2-3 branch via this commit: 13e4a75

This is not an issue on the main branch, where the explicit call to TaskInstance._run_finished_callback() has been removed from the DebugExecutor.

What you think should happen instead

No response

How to reproduce

No response

Operating System

Debian 11

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Would you like to fix it maybe @jochenott ? - You seem to know the code, and should be able to prepare a Pull Request.

[eladkal]

cc @o-nikolas related to #24357 ?

[o-nikolas]

related to #24357 ?

@eladkal, related yeah. My change depended on a previous change in main, but the previous change didn't make it into the 2.3 release branch it seems? From what I can tell everything has always been good in main and still seems to be. I'd say we can close this one?

[SwanseN21]

Apache Airflow version

2.3.3 (latest released)

What happened

The DebugExecutor runs the on_failure_callback twice.

This is due to TaskInstance._run_finished_callback() being called twice: once implicitly via the (new) call to ti.run(), once explicitly again:

https://github.com/apache/airflow/blob/62778bb8648515e9823ace2f48d9a89ec91a6803/airflow/executors/debug_executor.py#L79,L81

This came in to the airflow 2-3 branch via this commit: 13e4a75

This is not an issue on the main branch, where the explicit call to TaskInstance._run_finished_callback() has been removed from the DebugExecutor.

What you think should happen instead

No response

How to reproduce

No response

Operating System

Debian 11

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct