Fix flaky PhysicalFilesWatcher timeout when root is deleted and recreated#130967
Draft
svick wants to merge 1 commit into
Draft
Fix flaky PhysicalFilesWatcher timeout when root is deleted and recreated#130967svick wants to merge 1 commit into
svick wants to merge 1 commit into
Conversation
…ated TryEnableFileSystemWatcher decided whether to set up the root creation watcher solely based on FileSystemWatcher.EnableRaisingEvents. On Linux, when the watched root directory is deleted the runtime FileSystemWatcher tears down the inotify watch and queues an Error, but leaves EnableRaisingEvents == true. It is only reset once OnError runs TryDisableFileSystemWatcher. If a token was re-registered before that happened, TryEnableFileSystemWatcher saw EnableRaisingEvents == true and skipped setting up the PendingCreationWatcher, leaving a dead inotify watch bound to the deleted inode. Recreating the root never re-armed it, so the token never fired and the change token wait timed out (30s), causing the flaky failure in CreateFileChangeToken_RootDeletedAndRecreated_TokenFiresWhenFileCreated. Handle the case where the watcher still reports enabled but the root no longer exists by tearing down the stale watch and falling back to the root creation watcher. Fixes dotnet#129691 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ea806a13-89ae-46ef-a4d9-03a6db0a506e
|
Azure Pipelines: Successfully started running 3 pipeline(s). 12 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
|
Tagging subscribers to this area: @dotnet/area-extensions-filesystem |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens PhysicalFilesWatcher against a race where the underlying FileSystemWatcher can remain logically “enabled” while its OS-level watch has been torn down after the watched root directory is deleted (notably on Linux/inotify), causing subsequent re-registrations to miss root recreation and never observe the target file creation.
Changes:
- In
TryEnableFileSystemWatcher, detect the “enabled but root missing” state and proactively tear down the stale watch by settingEnableRaisingEvents = false. - Fall back to the existing root reappearance mechanism (
PendingCreationWatcher) by settingneedsRootWatcher = trueand marking_rootWasUnavailable = trueso the re-enable path performs the “gap” scan when the root comes back.
Member
Author
|
/azp run runtime-coreclr libraries-jitstress-random |
|
No pipelines are associated with this pull request. |
Member
Author
|
/azp list |
Member
Author
|
/azp run runtime-coreclr libraries-jitstress-random |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Member
Author
|
/azp run runtime-coreclr libraries-pgo, runtime-coreclr libraries-jitstress2-jitstressregs |
|
Azure Pipelines: Successfully started running 2 pipeline(s). |
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #129691.
Problem
PhysicalFilesWatcherTests.CreateFileChangeToken_RootDeletedAndRecreated_TokenFiresWhenFileCreatedintermittently fails with a 30sSystem.TimeoutException(the change token never fires), most frequently under jitstress/pgo legs on Linux.Root cause
TryEnableFileSystemWatcherdecided whether to set up thePendingCreationWatcher(the fallback that watches for the root directory to reappear) solely based onFileSystemWatcher.EnableRaisingEvents.On Linux, when the watched root directory is deleted, the runtime
FileSystemWatchertears down the inotify watch and queues anError, but does not resetEnableRaisingEvents— it keeps reportingtrue. The only thing that flips it tofalseisOnError→TryDisableFileSystemWatcher.The test deletes the root, then re-registers a token as soon as the initial token fires (from
OnError'sCancelAll). This is a race:TryEnableFileSystemWatcherruns afterOnError'sTryDisableFileSystemWatcher,EnableRaisingEventsisfalseand thePendingCreationWatcheris set up correctly.EnableRaisingEventsis stilltrue, so theif (!EnableRaisingEvents)block is skipped and noPendingCreationWatcheris created.TryDisableFileSystemWatcherthen sees the freshly re-added token and does not disable. The result is a dead inotify watch bound to the deleted inode: recreating the root never re-arms it, and the token never fires.Fix
In
TryEnableFileSystemWatcher, handle the case where the watcher still reports enabled but_rootno longer exists: tear down the stale watch (EnableRaisingEvents = false) and fall back to the root creation watcher. Since the test re-registers while the root is still deleted, this closes the race deterministically.Note
This pull request was created by GitHub Copilot.