Windows service crash when stop it

[analogrelay]

From @dospat07 on Wednesday, September 25, 2019 2:18:37 PM

To Reproduce

Steps to reproduce the behavior:

1. Using this version of ASP.NET Core 3.0 and Visual Studio 2019 version 16.3
2. Create new ASP.NET core API web application
3. Add Microsoft.Extensions.Hosting.WindowsServices
4. Change method CreateHostBuilder to

public static IHostBuilder CreateHostBuilder(string[] args) =>
          Host.CreateDefaultBuilder(args)
              .UseWindowsService()
              .ConfigureWebHostDefaults(webBuilder =>
              {
                  webBuilder.UseStartup<Startup>();
              });
  }

5. change appsettings.json to

{
     "Logging": {
        "EventLog": {
              "LogLevel": {
                     "Default": "Warning"      
                               }
                        }
                 },
       "AllowedHosts": "*"
}

6. create windows service via sc or powershell
7. start service
8. stop service
9. Error found in Application log in EventViewer
   Faulting application name: WebApplication3.exe, version: 1.0.0.0, time stamp: 0x5d7bb03a
   Faulting module name: ntdll.dll, version: 6.3.9600.19304, time stamp: 0x5c7f684f
   Exception code: 0xc0000374
   Fault offset: 0x00000000000f1cd0
   Faulting process ID: 0x3eb0
   Faulting application start time: 0x01d573a4eb992a85
   Faulting application path: C:\Users\xxx\Source\repos\WebApplication3\WebApplication3\bin\Debug\netcoreapp3.0\WebApplication3.exe
   Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
   Report ID: 4cb3564a-df98-11e9-82b2-082e5f215ab6
   Faulting package full name:
   Faulting package-relative application ID:

Copied from original issue: dotnet/extensions#2396

[analogrelay]

From @Tratcher on Wednesday, September 25, 2019 4:11:44 PM

Is there a dump you can share? Or another event log with additional error details?

[analogrelay]

From @dospat07 on Wednesday, September 25, 2019 5:04:10 PM

CrashDump
https://drive.google.com/file/d/1uPFjj5pRgZdhyM6mU_Pc1SFAcMhNYjTj/view?usp=sharing
Source
WebApplication3.zip

[analogrelay]

From @Tratcher on Tuesday, October 1, 2019 9:54:25 PM

Error from the dump:

Unhandled exception at 0x00007FF8CBB31CD0 (ntdll.dll) in WebApplication3.exe.16048.dmp: 0xC0000374: A heap has been corrupted (parameters: 0x00007FF8CBB6ED40). occurred

>	ntdll.dll!00007ff8cbb31cd0()	Unknown
 	ntdll.dll!00007ff8cbb34f12()	Unknown
 	ntdll.dll!00007ff8cbb35b10()	Unknown
 	ntdll.dll!00007ff8cbaea5ff()	Unknown
 	KERNELBASE.dll!00007ff8c8fb1972()	Unknown
 	System.Private.CoreLib.dll!00007ff87e7ad4ed()	Unknown
 	[Managed to Native Transition]	
 	System.Private.CoreLib.dll!System.Runtime.InteropServices.Marshal.FreeHGlobal(System.IntPtr hglobal)	Unknown
 	System.ServiceProcess.ServiceController.dll!System.ServiceProcess.ServiceBase.Dispose(bool disposing)	Unknown
 	Microsoft.Extensions.Hosting.WindowsServices.dll!Microsoft.Extensions.Hosting.WindowsServices.WindowsServiceLifetime.Dispose(bool disposing)	Unknown
 	System.ComponentModel.Primitives.dll!System.ComponentModel.Component.Dispose()	Unknown
 	Microsoft.Extensions.DependencyInjection.dll!Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.DisposeAsync()	Unknown
 	Microsoft.Extensions.Hosting.dll!Microsoft.Extensions.Hosting.Internal.Host.DisposeAsync()	Unknown
 	Microsoft.Extensions.Hosting.Abstractions.dll!Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(Microsoft.Extensions.Hosting.IHost host, System.Threading.CancellationToken token)	Unknown
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<RunAsync>d__4>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<RunAsync>d__4>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<RunAsync>d__4>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult(System.Threading.Tasks.VoidTaskResult result)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()	Unknown
 	[External Code]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<WaitForShutdownAsync>d__5>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<WaitForShutdownAsync>d__5>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<WaitForShutdownAsync>d__5>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult(System.Threading.Tasks.VoidTaskResult result)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()	Unknown
 	[External Code]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<StopAsync>d__10>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<StopAsync>d__10>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.Extensions.Hosting.Internal.Host.<StopAsync>d__10>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult(System.Threading.Tasks.VoidTaskResult result)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()	Unknown
 	[External Code]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Hosting.GenericWebHostService.<StopAsync>d__33>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Hosting.GenericWebHostService.<StopAsync>d__33>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Hosting.GenericWebHostService.<StopAsync>d__33>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.Threading.Tasks.VoidTaskResult>.TrySetResult(System.Threading.Tasks.VoidTaskResult result)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.SetResult()	Unknown
 	[External Code]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<StopAsync>d__22>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<StopAsync>d__22>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Threading.Tasks.VoidTaskResult>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer.<StopAsync>d__22>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<bool>.TrySetResult(bool result)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.SetResult(bool result)	Unknown
 	Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.CloseAllConnectionsAsync(System.Threading.CancellationToken token)	Unknown
 	[Resuming Async Method]	
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.<CloseAllConnectionsAsync>d__10>.ExecutionContextCallback(object s)	Unknown
 	System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.<CloseAllConnectionsAsync>d__10>.MoveNext(System.Threading.Thread threadPoolThread)	Unknown
 	System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder<bool>.AsyncStateMachineBox<Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.ConnectionManager.<CloseAllConnectionsAsync>d__10>.MoveNext()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox box, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task<System.__Canon>.TrySetResult(System.__Canon result)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.TaskFactory.CompleteOnInvokePromise.Invoke(System.Threading.Tasks.Task completingTask)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunOrQueueCompletionAction(System.Threading.Tasks.ITaskCompletionAction completionAction, bool allowInlining)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.RunContinuations(object continuationObject)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.FinishContinuations()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.TrySetResult()	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.Task.WhenAllPromise.Invoke(System.Threading.Tasks.Task completedTask)	Unknown
 	System.Private.CoreLib.dll!System.Threading.Tasks.CompletionActionInvoker.System.Threading.IThreadPoolWorkItem.Execute()	Unknown
 	System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch()	Unknown
 	[Native to Managed Transition]	
 	[External Code]	

[analogrelay]

From @anurse on Thursday, November 7, 2019 7:51:40 PM

Heap corruption seems unlikely to be due to ASP.NET Core. Adding @davidfowl.

[analogrelay]

From @anurse on Thursday, November 7, 2019 11:37:26 PM

@jkotas may be able to help here

[analogrelay]

From @jkotas on Friday, November 8, 2019 12:26:50 AM

System.ServiceProcess.ServiceBase.Dispose is not thread safe. This crash is caused by System.ServiceProcess.ServiceBase.Dispose being called on multiple threads at the same time.

.NET Framework implementation of System.ServiceProcess.ServiceBase has the same issue. This is not .NET Core regression.

This should be transferred to dotnet/corefx to area-System.ServiceProcess.

[stephentoub]

This crash is caused by System.ServiceProcess.ServiceBase.Dispose being called on multiple threads at the same time.
This should be transferred to dotnet/corefx to area-System.ServiceProcess.

@anurse, who's calling it concurrently on multiple threads? The thread above shows at least one of the callers to be from Microsoft.Extensions.Hosting. Isn't the issue here that the class is being misused rather than it being a bug in the class? Many Dispose methods across .NET aren't thread-safe and could exhibit bad behaviors if invoked concurrently.

[jkotas]

Isn't the issue here that the class is being misused

Agree. I am re-activating dotnet/extensions#2396.

(We should still fix System.ServiceProcess.ServiceBase to stop corrupting unmanaged heap on misuse.)

[jkotas]

I do not have the rights to reactive dotnet/extensions#2396. @anurse Could you please reactivate it?

[stephentoub]

We should still fix System.ServiceProcess.ServiceBase to stop corrupting unmanaged heap on misuse.

Agreed.

[analogrelay]

All sounds good. Glad we're fixing the double-free, and we'll look in to the double-dispose over on our side. Then we'll be double-good ;).