Release mutex when facing AbandonedMutexException

[daxian-dbw]

PR Summary

Fix #2742

When getting a AbandonedMutexException, the current thread actually owns the mutex at that point, and thus it must release it. Otherwise, the mutex object will be kept held by that thread, resulting in all WaitOne calls from other PowerShell processes always time out.

Quoted from the sample code for AbandonedMutexException

// Whether or not the exception was thrown, the current
// thread owns the mutex, and must release it.

I have verified that when not releasing mutex in case of AbandonedMutexException, another thread calling WaitOne on the mutex will always time out (see the output messages below).

In the sample program, the mutex is not released in the exception handler when AbandonedMutexException is caught in the main thread. The subsequent retry of the WaitOne call in main thread succeeded, but even though ReleaseMutex is called for that successful WaitOne call, the main thread is still holding the mutex, because it didn't release the mutex when AbandonedMutexException was caught.

Then when trying to call WaitOne on the same mutex object from a different thread, the call always times out.

PS:28> dotnet run
< 'AbandonMutex' thread starts:
    Thread exits without releasing the mutexes.
'AbandonMutex' thread ends. >

< Main thread:
    AbandonedMutexException from _orphan1.WaitOne.
    Retry without release mutex.
    ---- Re-try 1:
    WaitOne call succeeded.
    Release mutex
Main thread starts 'TryAquaireMutex' thread and wait. >

< 'TryAquaireMutex' thread starts:
    Timed out. Re-try 1
    Timed out. Re-try 2
    Timed out. Re-try 3
    Timed out. Re-try 4
    Timed out. Re-try 5
'TryAquaireMutex' thread ends. >

< Main thread continue:
    retry counts: 1
    Call ReleaseMutex
Main thread ends. >

---

So the fix in PSReadLine is to call ReleaseMutex() in the exception handler of AbandonedMutexException, so that we don't miss a release and indefinitely hold the mutex without knowing. TBH, I don't know how the mutex could be abandoned by PSReadLine. It seems impossible by looking at the code, but I guess it could happen for some random reasons. It would be nice to have diagnostics telemetry in PSReadLine to help confirm that.

PR Checklist

• PR has a meaningful title
  • Use the present tense and imperative mood when describing your changes
• Summarized changes
• Make sure you've added one or more new tests
• Make sure you've tested these changes in terminals that PowerShell is commonly used in (i.e. conhost.exe, Windows Terminal, Visual Studio Code Integrated Terminal, etc.)
• User-facing changes
  • Not Applicable
  • OR
  • Documentation needed at PowerShell-Docs
    • Doc Issue filed:

Microsoft Reviewers: Open in CodeFlow

[ghost]

🎉 v2.2.0-beta4 has been released which incorporates this pull request. 🎉