Respect retries when a deferrable trigger ends a task with TaskFailedEvent#69821
Open
hkc-8010 wants to merge 1 commit into
Open
Respect retries when a deferrable trigger ends a task with TaskFailedEvent#69821hkc-8010 wants to merge 1 commit into
hkc-8010 wants to merge 1 commit into
Conversation
…Event When a deferrable operator's trigger yields a terminal TaskFailedEvent, the task was always marked failed and on_failure_callback ran, even with retries remaining. A worker-side failure with retries left instead goes up_for_retry and runs on_retry_callback. Route trigger-emitted failures through TaskInstance.handle_failure (the path the scheduler has used since apache#56586) so retry-eligibility is respected and the callback runs on_retry_callback vs on_failure_callback accordingly. closes: apache#69819 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
hkc-8010
force-pushed
the
fix/deferred-task-failed-event-respects-retries
branch
from
4017af8 to
5dfb1ce
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a deferrable operator's trigger yields a terminal
TaskFailedEvent, the task was always markedfailedand itson_failure_callbackran, even when the task still had retries remaining. A worker-side failure with retries left instead goesup_for_retryand runson_retry_callback. This makes trigger-driven failures behave the same way.The
BaseTaskEndEventhandler inairflow-core/src/airflow/models/trigger.pynow, for aFAILEDevent, loads the serialized task and routes throughTaskInstance.handle_failure(), the same path the scheduler executor-event handling has used since #56586.handle_failuresetsup_for_retryorfailedbased on retry-eligibility and fires theon_task_instance_failedlistener, failure metrics, and a Log entry. The callback request now carries the resolvedtask_callback_typeso the DAG processor runson_retry_callbackvson_failure_callbackcorrectly.TaskSuccessEventandTaskSkippedEventare unchanged.closes: #69819
Changes
airflow-core/src/airflow/models/trigger.py: route trigger-emitted failures throughhandle_failure, thread the retry-awaretask_callback_typeinto the callback, and skip the xcom push on the retry path.airflow-core/tests/unit/models/test_trigger.py: new parametrized test covering the retry-eligible case (up_for_retry+on_retry_callback+on_task_instance_failedlistener) and the exhausted case (failed+on_failure_callback); the existing end-event test now sets a realisticstart_date(a deferred task has already run).airflow-core/newsfragments/69821.bugfix.rst.Testing
pytest airflow-core/tests/unit/models/test_trigger.py(39 passed).main(the task staysfailedand the callback is routed as a failure) and passes with this change.This change was written with AI assistance (Claude); the author reviewed the diff and reproduction.