Guard against exceptions in Learning transport that can happen due to competing consumer starting

[danielmarbach]

While migrating the monitoring demo to the learning transport we found that in some scenarios with competing consumers the endpoint might fail to start with the following exception

Unhandled Exception: System.IO.IOException: The process cannot access the file '432e0ce9-851b-46cb-b156-16d390325ff9.metadata.out' because it is being used by another process.
  at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
  at System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
  at NServiceBus.DirectoryBasedTransaction.RecoverPartiallyCompletedTransactions(String basePath, String pendingDirName, String committedDirName)
  at NServiceBus.LearningTransportMessagePump.Start(PushRuntimeSettings limitations)
  at NServiceBus.TransportReceiver.Start()
  at NServiceBus.ReceiveComponent.Start()
  at NServiceBus.StartableEndpoint.<Start>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at NServiceBus.Endpoint.<Start>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at Sales.Program.<Main>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  at Sales.Program.<Main>(String[] args)

After doing short investigation it looks that LearningTransport does not consider competing consumers during start-up when it cleans the unfinished transactions. What is probably happening is that he running endpoint has an opened transaction that is consider by new starting instance as unfished pending. In such case the new one tries to clean the transaction and delete message folder which fails as it is locked by other instance.

Particular/MonitoringDemo#51 (comment)

[andreasohlund]

Should we add retries similar to https://github.com/Particular/NServiceBus/blob/develop/src/NServiceBus.Core/Transports/Learning/AsyncDirectory.cs ?

[tmasternak]

@andreasohlund I've moved the pending transaction recovery to a background process. If there are any errors the hope is those will be fixed on the next run.

@danielmarbach I've tested this version with MonitoringDemo and could not break it.

[tmasternak]

I've added two more changes to handle exceptions observed while testing:

• IOException handling is not enough. Switched to Exception instead. Here is a sample of UnauthroizedAccessException being thrown

Unhandled Exception: System.UnauthorizedAccessException: Access to the path is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
   at NServiceBus.DirectoryBasedTransaction.RecoverCommitted()
   at NServiceBus.DirectoryBasedTransaction.RecoverPartiallyCompletedTransactions(String basePath, String pendingDirName, String committedDirName)
   at NServiceBus.LearningTransportMessagePump.RecoverPendingTransactions()
   at NServiceBus.LearningTransportMessagePump.Start(PushRuntimeSettings limitations)
   at NServiceBus.TransportReceiver.Start()
   at NServiceBus.ReceiveComponent.Start()
   at NServiceBus.StartableEndpoint.<Start>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at NServiceBus.Endpoint.<Start>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sales.Program.<Main>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Sales.Program.<Main>(String[] args)

• Commit recovery can crash on directory enumeration just as pending recovery so the try-catch has been extended.

After those changes I could not get the monitoring demo to crash.

[danielmarbach]

I patched the monitoring demo with this latest version and let it run with high throughput mode with scaleout sales endpoint for over an hour. Seems to be fine and we are no longer running into the problem that the endpoints get stopped to due CriticalError.Raise and the competing recover problems are also gone

[bording]

Changes LGTM, but we still need to figure out what branch this needs to go in.

[bording]

I've rebased the branch and opened #5309 to have this go against master instead, so I'm going to close this.