FileSystemWatcher SymbolicLink tests failing after inotify refactor (#117148)

[lewing]

Summary

Multiple FileSystemWatcher SymbolicLink tests have started failing on Linux since the inotify refactor in #117148 (merged Jan 26, 2026). The failures appear as missed Created events with the message "Created event did not occur as expected".

Suspected Root Cause

PR #117148 refactored the Linux FileSystemWatcher to use a single shared inotify instance. There appears to be a race condition in Watcher.Start() (FileSystemWatcher.Linux.cs) where Start() returns before the ProcessEvents background thread is guaranteed to be reading from the inotify file descriptor:

StartThread()           // spawn background thread
CreateRootWatch()       // register inotify watch with kernel
_inotify.AddWatcher()   // register with shared instance
DequeueEvents()         // async fire-and-forget
_emitEvents = true
// Start() returns — but ProcessEvents may not be reading yet

If a filesystem event occurs in this gap (between Start() returning and ProcessEvents entering its read() loop), the event can be missed or delayed past the test timeout.

Affected Tests

• SymbolicLink_Changed_Tests.FileSystemWatcher_SymbolicLink_TargetsDirectory_Create — failing on linux arm32 jitstress (System.IO.Tests.Directory_Create_Tests.FileSystemWatcher_Directory_Create_InNestedDirectory fails with "Created event did not occur as expected" #103630 false match)
• SymbolicLink_Changed_Tests.FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories — consistently failing, disabled in Disable FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories #124849

The _IncludeSubdirectories variant fails more consistently because it requires multiple watch registrations, widening the race window.

Timeline

• Jan 26: PR FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148 merged (inotify refactor)
• Feb 20: FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories failed with missed event #124677 filed — _IncludeSubdirectories variant failing
• Feb 25: FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories failing on multiple OSes #124847 / Disable FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories #124849 — test disabled
• Mar 17: Non-_IncludeSubdirectories variant appearing in CI, incorrectly attributed to System.IO.Tests.Directory_Create_Tests.FileSystemWatcher_Directory_Create_InNestedDirectory fails with "Created event did not occur as expected" #103630 via overly broad error pattern match

Environment

Failures observed on:

• Linux ARM32 (Debian 13 on AzureLinux 3 Arm64 cross-arch) under jitstress
• Multiple Linux x64 configurations (crossgen2, libraries-jitstress)

JitStress and slower hardware (arm32) widen the race window, making failures more frequent.

Related Issues

• FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories failed with missed event #124677 — _IncludeSubdirectories Known Build Error (27 hits in first month)
• FileSystemWatcher_SymbolicLink_TargetsDirectory_Create_IncludeSubdirectories failing on multiple OSes #124847 — same, disabled-test tracking
• System.IO.Tests.Directory_Create_Tests.FileSystemWatcher_Directory_Create_InNestedDirectory fails with "Created event did not occur as expected" #103630 — old Windows FSW flakiness, now falsely matching this Linux failure due to broad error pattern

Possible Fix

Add a synchronization point in Watcher.Start() that ensures the ProcessEvents thread is blocked on read() before Start() returns — for example a ManualResetEventSlim signaled by the processing thread once it enters the read loop.

/cc @tmds @adamsitnik @jozkee

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[lewing]

we're trying bumping the timeout since the kernel should be handling the buffering #125744