Stop failing reschedule-mode sensors on a stale executor success - #71194
Draft
rjgoyln wants to merge 1 commit into
Draft
Stop failing reschedule-mode sensors on a stale executor success#71194rjgoyln wants to merge 1 commit into
rjgoyln wants to merge 1 commit into
Conversation
A sensor running with mode="reschedule" was intermittently marked failed when the executor success for one poke reached the scheduler after the task instance had already been put back for the next poke. The existing guards for this race only cover the defer path: a reschedule exit leaves next_method unset and does not bump try_number, so none of them applied and the event was treated as an external kill.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A sensor running with
mode="reschedule"is intermittently marked failed when the executor success for one poke reaches the scheduler after the task instance has already gone back toscheduled(orqueued) for the next poke. The scheduler reads the late event as a task that finished without recording its own state, and fails it:The existing guards for this race all assume the defer path. A reschedule exit leaves
next_methodunset and keeps the sametry_number, so the executor key still matches and none of them apply. A reschedule row for the current try is the signal that separates the two cases: it says the worker exitedup_for_reschedulerather than being killed. Anything with no such row still takes the externally-killed branch.It is resolved once per event batch alongside the bulk task-instance fetch, so the per-event path gains no query.
closes: #71172
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines