Fix Nanny race conditions when worker fails to start (flaky test_failure_during_worker_initialization)#9313
Merged
crusaderky merged 11 commits intoJul 2, 2026
Conversation
Contributor
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 40 files ± 0 40 suites ±0 14h 54m 18s ⏱️ + 38m 53s For more details on these failures and errors, see this check. Results for commit a571c4a. ± Comparison against base commit 2d43f7f. This pull request removes 2 tests.♻️ This comment has been updated with latest results. |
crusaderky
force-pushed
the
test_failure_during_worker_initialization
branch
2 times, most recently
from
016ac23 to
a86d6d3
Compare
When a worker subprocess fails during initial startup (e.g. an invalid kwarg), it reports the failure via init_result_q and then exits gracefully. On slower machines this raced with both the process-exit callback and the Nanny startup teardown, producing several intermittent failures: a spurious "Restarting worker", a 30s hang, or a leaked QueueFeederThread. Three coordinated fixes in nanny.py: - Nanny._on_worker_exit: add Status.init to the restart skip-set, so a worker process exit during initial startup is not mistaken for a transient post-startup crash and does not trigger a restart. (This matches the skip-set already used just above for unregistering.) - WorkerProcess.start: only transition starting -> failed. A concurrent mark_stopped() (fired by the process-exit callback) may already have moved the state to stopped and released resources; clobbering it with failed left a failed-but-released state that made kill() raise midway through Nanny.close(), leaving the Nanny in Status.closing so the next close() hung on finished() until the test timed out. - WorkerProcess.start: drop the eager terminate() on startup failure. The subprocess is already shutting down after reporting the failure; a SIGTERM races that shutdown and, if it lands while the subprocess holds a multiprocessing queue lock, deadlocks the parent-side queue feeder thread. Any lingering process is still reaped gracefully by the following Nanny.close() -> WorkerProcess.kill(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
crusaderky
force-pushed
the
test_failure_during_worker_initialization
branch
from
76454cd to
2376b46
Compare
The subprocess reports its startup failure via init_result_q and then exits on its own. While WorkerProcess.start()'s exception handler awaits terminate(), the process-exit callback can fire mark_stopped(), which sets Status.stopped and releases init_result_q, child_stop_q and process. Unconditionally overwriting that with Status.failed made the following Nanny.close() -> WorkerProcess.kill() skip its Status.stopped early exit and crash with an AttributeError on the released child_stop_q, aborting Nanny.close() midway with the Nanny stuck in Status.closing. The second close() call from Server.start_unsafe's exception handler then waited on finished() forever, hanging the test until its 30s timeout. Guard the terminate() call (the process may already be released) and only transition starting -> failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
crusaderky
changed the title
test_failure_during_worker_initializationtest_failure_during_worker_initialization)
crusaderky
added
bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix race conditions triggered when the worker fails to start (eg. in the test, due to a bad kwarg) and: