[Windows] FileSystemWatcher causes SOH fragmentation from memory leak of pinned 8K byte[] buffers

[subbucse]

Description

• Please review illustrative PR: [Windows] Illustration of fix in FileSystemWatcher to prevent SOH fragmentation from memory leak of pinned 8K byte[] buffers #116613
• Illustration with comments of race condition between garbage collection of FileSystemWatcher instance and directory changes callback with errorCode = ERROR_OPERATION_ABORTED on IOCP thread, when FileSystemWatcher is disposed before releasing its last reference.
• When GC wins the race, we are leaving the door open to leaking the pinned 8k byte[] on SOH which over time will cause heavy heap fragmentation preventing heap compaction and affecting allocation and GC performance.
• Placeholder for fix with cleanup code when we detect that the WeakRef in the AsyncState in the IO callback has been released.
• Suggestion to move pinned buffer allocation to POH at time of alloc instead of alloc on SOH and pinning soon after, so that fragmentation is avoided even when FileSystemWatcher is kept alive.
• Attached a sample program to produce the memory leak and also if needed (to microsoft internal maintainers) I can produce a process dump with full heaps from our production service which shows the leak.

Reproduction Steps

• Please review illustrative PR: [Windows] Illustration of fix in FileSystemWatcher to prevent SOH fragmentation from memory leak of pinned 8K byte[] buffers #116613
• Illustrative sample program in included in the PR to reproduce the heap leak in FileSystemWatcher in the PR as FileSystemWatcherHeapLeak.csproj
• Please run with strictest input parameters as disposeProbability = 100, maxGeneration = 2, compacting = true, and heap will start growing from leaked 8K byte[] buffers eventually reaching ~2 GB on my devbox over 3-4 hours.
• If you relax the parameters, say disposeProbability < 100, maxGeneration = 0/1, compacting = false, you will only increase the rate at which heap leak shows up

Expected behavior

• There should be no leak of 8k pinned byte[] buffers over time in the sample program
• The pinned buffers ought to not be causing fragmentation in the SOH, they ought to utilize the POH and be pinned at allocation time.

Actual behavior

• Heap will start growing from leaked 8K byte[] buffers in the sample program eventually reaching ~2 GB on my devbox over 3-4 hours.
• The pinned buffers end up causing heavy fragmentation in the SOH and prevent compaction, increasing allocation costs, decreasing GC performance.

Regression?

Don't think so.

Known Workarounds

No response

Configuration

• MsCorlib and System.IO.FileSystem.Watcher module version 8.0.224.6711
• Windows, exact version can be found in !di command in process dump I can share with dotnet engineers.
• Not sure if it occurs on Linux too

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

[subbucse]

Please forgive the duplication from illustrative PR https://github.com/dotnet/runtime/pull/116613/files .

@stephentoub You mentioned in our teams chat that there was explicit intent to avoid pinning FSW buffers on POH because the pinning is supposed to be temporary. Did you mean that you expect the pinned buffer to be unpinned and given up for GC after every single IO callback comes in from every async overlapped call to ReadDirectoryChangesW ?

If so, I understand not using POH, because the pinning is highly transient, but that's not what's happening now. As you see below, we only call state.ThreadPoolBinding.FreeNativeOverlapped(overlappedPointer) which will internally call AsyncState.PreAllocatedOverlapped.Release(), and NOT AsyncState.PreAllocatedOverlapped.Dispose() which is what ultimately unpins and frees up the pinned 8K byte[] buffer.

So if the intent is to keep the buffer pinned until end of lifetime of FSW, then shouldn't we consider using POH since that timeframe could be long?

[stephentoub]

I responded on that PR.