Fix FileSystemWatcher test flakiness on Windows and re-enable 86 disabled tests

[danmoseley]

Categories of failures

The FSW test failures on Windows fall into three categories:

1. Timeout too short for callback latency — Under thread pool pressure (typical in CI), ReadDirectoryChangesW callbacks arrive late. The 1000ms WaitForExpectedEventTimeout expires before the event is delivered. This is the dominant failure mode: "<EventType> event did not occur as expected".

2. Stale state between retries — Some tests create files/directories in action() but never clean up. When ExpectEvent retries (re-creating the watcher and re-running the action), the second attempt fails because the file already exists, or the watcher fires on leftover state rather than the fresh action.

3. ExpectNoEvent using wrong timeout — ExpectNoEvent was hardcoded to use WaitForExpectedEventTimeout (1000ms) instead of WaitForUnexpectedEventTimeout (500ms), making negative tests take 2x longer than intended and potentially masking timing issues.

Fixes in this PR (and why each)

No product changes. All changes are in the test utility (FileSystemWatcherTest.cs) and individual test files.

1. Progressive retry timeouts — ExpectEvent now multiplies WaitForExpectedEventTimeout by the attempt number (1x, 2x, 3x). This directly addresses category 1: if the first attempt's 1000ms is too short under load, the second attempt waits 2000ms and the third waits 3000ms, giving the callback time to arrive without penalizing the common fast case.

2. Increased SubsequentExpectedWait (10ms to 500ms) — Tests that check multiple event types (e.g., Changed + Created) waited only 10ms for the second event type after catching the first. Under load, the second event arrives later. 500ms provides adequate margin.

3. Settling delay after enabling watcher (50ms Thread.Sleep) — Added in ExecuteAndVerifyEvents, ExpectEvents, and TryErrorEvent after setting EnableRaisingEvents = true and before running the test action. While Windows registers the watch synchronously, this guards against edge cases on all platforms.

4. Cleanup in Create event tests — File.Create.cs and Directory.Create.cs tests now delete the created file/directory in a finally block, so retries start from a clean state (category 2).

5. ExpectNoEvent timeout fix — Changed to use WaitForUnexpectedEventTimeout as intended (category 3).

6. Re-enabled 86 tests previously disabled by 25 [ActiveIssue] annotations (11 class-level, 14 method-level) referencing Failing test due to no detected IO events in 'System.IO.Tests.FileSystemWatcherTest.ExecuteAndVerifyEvents' #103584 on Windows. Two tests (File_Move_With_Set_Environment_Variable, File_Move_With_Unset_Environment_Variable) are disabled against a new issue for a product-level bug unrelated to timing.

Validation steps

Tested on Intel i9-14900K (24 cores / 32 threads), 64 GB RAM, Windows 11. Tests run on both NTFS and ReFS (dev drive).

Controlled callback delay (proving the fix addresses the root cause):

• Injected a temporary busy-spin delay into ReadDirectoryChangesCallback to deterministically reproduce the late-event-delivery condition that occurs under thread pool starvation in CI
• At 900ms delay, baseline (no fixes): 114 event failures across 20 runs (100% failure rate)
• At 900ms delay, with fixes: 0 event failures across 20 runs

Stress testing with fixes applied (proving stability under realistic CI-like load):

• Concurrent stress generator running alongside tests: 32 CPU-saturating workers (one per logical processor) doing tight math loops, 8 IO workers (rapid file create/write/delete cycles with 1-64KB random data), and continuous thread pool flooding (~500 short work items queued every 10ms) to simulate thread pool contention
• Under this load, ran 20 sequential iterations on each filesystem: 0 flaky failures on NTFS, 0 on ReFS
• 3 parallel test instances running simultaneously under stress: 0 flaky failures

Unchanged

• Linux/macOS — Separate issues exist (FileSystemWatcher SymbolicLink tests failing after inotify refactor (#117148) #125737 Linux inotify race after FileSystemWatcher.Linux: use a single inotify instance and refactor watch tracking. #117148 refactor; System.IO.Tests.File_Move_Tests.File_Move_Multiple_From_Watched_To_Unwatched_Mac test failure #125018 macOS duplicate FSEvents). These have different root causes and are not addressed here.
• Product code — No changes to FileSystemWatcher implementation. The callback delivery latency is fundamentally an OS/thread-pool scheduling issue, not a product bug. The fix is making tests tolerant of realistic delivery variance.
• Issue FileSystemWatcher_DirectorySymbolicLink_TargetsFile_Fails #53366 (SymbolicLink / InternalBufferSize tests) — Remains disabled on all platforms; this is a separate product-level issue.

Fixes #103630
Fixes #103584

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR targets Windows flakiness in System.IO.FileSystem.Watcher tests by increasing tolerance to callback latency and retry-related stale state, and then re-enables a large set of tests previously disabled on Windows.

Changes:

• Adjusts FileSystemWatcherTest timeouts/retry behavior (including progressive timeouts) and fixes ExpectNoEvent default timeout usage.
• Adds a small post-EnableRaisingEvents settling delay to reduce timing sensitivity across platforms.
• Re-enables many tests previously disabled via [ActiveIssue], and adds cleanup in some create-related tests to keep retries idempotent.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs	Updates core test helper timeouts/retry logic and adds settling delays.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.unit.cs	Removes Windows [ActiveIssue] suppressions and adds cleanup for create-related retry safety.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.cs	Re-enables a previously disabled init/resume test on Windows.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.SymbolicLink.cs	Re-maps Windows ActiveIssue annotations for specific symlink tests.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.InternalBufferSize.cs	Removes Windows [ActiveIssue] suppression for internal buffer size tests.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.NotifyFilter.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Move.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Delete.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Create.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.File.Changed.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.NotifyFilter.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Move.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Delete.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Create.cs	Removes Windows [ActiveIssue] suppression at the class level.
src/libraries/System.IO.FileSystem.Watcher/tests/FileSystemWatcher.Directory.Changed.cs	Removes Windows [ActiveIssue] suppression at the class level.

Comments suppressed due to low confidence (1)

src/libraries/System.IO.FileSystem.Watcher/tests/Utility/FileSystemWatcherTest.cs:286

• cleanup() is executed after the assertion. If the assertion fails (i.e., an unexpected event did occur), cleanup won’t run, which can leave state behind for the outer retry wrapper and lead to follow-on failures. Run cleanup in a finally so it executes regardless of assertion outcome.

            bool result = ExecuteAndVerifyEvents(watcher, unExpectedEvents, action, false, expectedPath == null ? null : new string[] { expectedPath }, timeout);
            Assert.False(result, "Expected Event occurred");

            if (cleanup != null)
                cleanup();

[Copilot]

The comment says subsequent retries "double" the timeout, but effectiveTimeout = timeout * attemptsCompleted scales linearly (1x, 2x, 3x). Update the comment to match the actual behavior (or change the implementation if exponential backoff is intended).

Suggested change

	// for fast failure in normal conditions. Subsequent attempts double the timeout to tolerate
	// transient delays (thread pool starvation, slow CI machines, etc.).
	// for fast failure in normal conditions. Subsequent attempts increase the timeout linearly
	// with the attempt count to tolerate transient delays (thread pool starvation, slow CI machines, etc.).

[Copilot]

This retry path recreates FileSystemWatcher instances, but the helper doesn’t dispose watchers created during retries (and it doesn’t return the recreated watcher to the caller). In flaky runs this can leak OS resources/handles and make subsequent retries/tests less reliable. Consider disposing any temporary recreated watchers (and/or redesigning the helper to surface the recreated watcher to the caller).

[Copilot]

cleanup() is invoked after ExecuteAndVerifyEvents, but if ExecuteAndVerifyEvents throws (e.g., on the final attempt where it asserts), cleanup won’t run. To keep retries reliable and avoid leaving stale filesystem state behind, run cleanup in a finally around the attempt execution.

Suggested change

	result = ExecuteAndVerifyEvents(newWatcher, expectedEvents, action, attemptsCompleted == attempts, expectedPaths, effectiveTimeout);
	if (cleanup != null)
	cleanup();
	try
	{
	result = ExecuteAndVerifyEvents(newWatcher, expectedEvents, action, attemptsCompleted == attempts, expectedPaths, effectiveTimeout);
	}
	finally
	{
	cleanup?.Invoke();
	}

[Copilot]

The default timeout here was changed to WaitForUnexpectedEventTimeout, but that constant is currently 150ms. Given the motivation of tolerating delayed callback delivery, 150ms may be too short to reliably detect “unexpected” events that arrive late under load; consider increasing the default/constant (or document why 150ms is sufficient).

[Copilot]

ExecuteAndVerifyEvents sets up handlers and enables EnableRaisingEvents, but there’s no try/finally around action() and the subsequent waits. If action() throws, handlers won’t be detached and EnableRaisingEvents won’t be reset, which can leak state into retries/other tests. Wrap the body in try/finally to always detach handlers and restore EnableRaisingEvents.

[Copilot]

In TryErrorEvent, the attempt adds an Error handler each time but never removes it, and the recreated watchers from retries aren’t disposed. Over multiple retries this can accumulate handlers and leak watcher resources. Consider detaching the handler and disposing any watcher instances created during retries (and do this in a finally so it also runs when action() throws).