Error -4082 EBUSY resource busy or locked

[tstar]

I have minimal MVC application with one slow method Index in Home controller. I start dnx web and in browser run http://localhost:5000/. Then I press and hold F5 button for refresh page. After some time my page is not responded. After I trying to stop Kestrel via Ctrl+C I get error:

C:\WebApplication1\src\WebApplication1>dnx web
Hosting environment: Production
Application base path: C:\WebApplication1\src\WebApplication1
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
System.Threading.ThreadAbortException: The stream was in the process interruption.
   в System.Threading.Monitor.Enter(Object obj)
   в Microsoft.AspNetCore.Server.Kestrel.Http.Connection.Abort()
   в Microsoft.AspNetCore.Server.Kestrel.Http.SocketOutput.OnWriteCompleted(WriteContext writeContext)
   в Microsoft.AspNetCore.Server.Kestrel.Http.SocketOutput.WriteContext.CompleteWithContextLock()
   в Microsoft.AspNetCore.Server.Kestrel.Networking.UvWriteReq.UvWriteCb(IntPtr ptr, Int32 status)
   в Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.NativeMethods.uv_run(UvLoopHandle handle, Int32 mode)
   в Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.run(UvLoopHandle handle, Int32 mode)
   в Microsoft.AspNetCore.Server.Kestrel.KestrelThread.ThreadStart(Object parameter)
--- The end of the stack trace from the previous location where the exception occurred ---
   в Microsoft.AspNetCore.Server.Kestrel.KestrelThread.Stop(TimeSpan timeout)
   в Microsoft.AspNetCore.Server.Kestrel.KestrelEngine.Dispose()
   в Microsoft.AspNetCore.Server.Kestrel.KestrelServer.Dispose()
   в Microsoft.AspNetCore.Hosting.Internal.WebHost.Dispose()
   в Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host, CancellationToken token, String shutdownMessage)
   в Microsoft.AspNetCore.Hosting.WebHostExtensions.Run(IWebHost host)
   в WebApplication1.Startup.Main(String[] args) в C:\WebApplication1\src\WebApplication1\Startup.cs:строка 40
--- The end of the stack trace from the previous location where the exception occurred ---
   в System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   в Microsoft.Dnx.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider serviceProvider)
   в System.Threading.Tasks.Task`1.InnerInvoke()
   в System.Threading.Tasks.Task.Execute()

An unhandled exception: Microsoft.AspNetCore.Server.Kestrel.Networking.UvException: Error -4082 EBUSY resource busy or locked
   в Microsoft.AspNetCore.Server.Kestrel.Networking.Libuv.Check(Int32 statusCode)
   в Microsoft.AspNetCore.Server.Kestrel.Networking.UvLoopHandle.ReleaseHandle()
   в System.Runtime.InteropServices.SafeHandle.InternalFinalize()
   в System.Runtime.InteropServices.SafeHandle.Finalize()

My Index method:

public IActionResult Index()
{
    Thread.Sleep(200);
    return View();
}

My project.json file:

{
  "version": "1.0.0-*",
  "compilationOptions": {
    "emitEntryPoint": true
  },

  "dependencies": {
    "Microsoft.NETCore.Platforms": "1.0.1-*",
    "Microsoft.AspNetCore.Server.Kestrel": "1.0.0-rc2-20121",
    "Microsoft.AspNetCore.IISPlatformHandler": "1.0.0-rc2-20121",
    "Microsoft.AspNetCore.Mvc": "1.0.0-rc2-20121",
    "Microsoft.AspNetCore.Mvc.Dnx": "1.0.0-rc2-20121"
  },

  "commands": {
    "web": "WebApplication1"
  },

  "frameworks": {
    "net451": { }
  },

  "exclude": [
    "wwwroot",
    "node_modules"
  ],
  "publishExclude": [
    "**.user",
    "**.vspscc"
  ]
}

[benaadams]

Try with await Task.Delay(200); rather than Thread.Sleep(200); and see what the results are. The latter is locking up a thread for every request which would be the equivalent of a heavy cpu bound task; otherwise it should be async.

[tstar]

I'll try. But I delay introduced to mimic the actual code from my project, which is carried out quickly, but leads to the same error with frequent refresh the page.

[tstar]

I Tried. Anyway, the problem persists. The new code of my method:

public async Task<IActionResult> Index()
{
    await Task.Delay(200);
    return View();
}

Is it correct?

[halter73]

This is an interesting issue. It seems to me that Connection.Abort() deadlocked waiting on the _stateLock on a uv thread.

I don't understand how this could happen, since the _stateLock should never be acquired for long and user code should not run with the lock.

@tstar If you're using VS, can you disable Just My Code, attach the debugger when your page stops responding, and show us your full Parallel Stacks window?

Thanks.

[halter73]

@tstar Your new code is what @benaadams was suggesting, but I would like you to attach the debugger with the old code if the issue doesn't reproduce with Task.Delay.

[tstar]

Apparently I have no experience of how to attach a debugger to the Kestrel. If I just run my code from visual studio debug mode, my web page also stop responding, and Kestrel console does not respond to Ctrl + C. Sorry.

[tstar]

If I start MVC application via IIS Express command, then web page frozen without any errors, because in this mode I haven't Kestrel console. I think, in this mode my site also work on Kestrel, because in Startup.cs I have code:

public static void Main(string[] args)
{
    var host = new WebHostBuilder()
        .UseDefaultConfiguration(args)
        .UseServer("Microsoft.AspNetCore.Server.Kestrel")
        .UseIISPlatformHandlerUrl()
        .UseStartup<Startup>()
        .Build();
    host.Run();
}

I'm right?

[halter73]

@tstar your description was enough to allow me to reproduce the issue and I now see the deadlock that is causing this. Great find! Thanks.

The deadlock occurs because Connection.Abort is called with the SocketOutput._contextLock acquired in SocketOutput.OnWriteCompleted. At the same time Connection.End has already been called and has already acquired the Connection._stateLock that Connection.Abort is trying to acquire. Unfortunately, Connection.End tries to acquire the _contextLock via SocketOutput.Write.

tl;dr. Both SocketOutput.OnWriteCompleted and Connection.End try to acquire the same two locks in the opposite order and fail predictably. SocketOutput.OnWriteCompleted is called on the libuv event loop which completely stops all request processing, not just the connection that raced and deadlocked.

I think the simplest solution is moving the call to System.Threading.ThreadPool.QueueUserWorkItem to the top level of Connection.Abort and acquiring the _stateLock on the threadpool thread.

[benaadams]

Parallel Stacks? What is this witchery!

[cesarblum]

Moving the call to QueueUserWorkItem would indeed eliminate the deadlock but I'm wondering about the proper way to set _connectionState without the lock. Setting it inside the lambda doesn't seem like the proper thing to do. Ideally I should be able to call Interlocked.CompareExchange(ref _connectionState, ConnectionState.ToDisconnect, ConnectionState.CreatingFrame) and if the return is != ConnectionState.CreatingFrame I would queue the callback and aquite _stateLock in it. However, Interlocked.ComparedExchange doesn't work with enum types. I could change _connectionState to int though, despite the ugliness. Any alternative ideas?

[cesarblum]

Or I could TryLock _stateLock and if it fails I would enqueue the work item acquiring the lock inside it.

[cesarblum]

I misread what @halter73 meant. I thought he meant to lock (_stateLock) inside the enqueued work item, but that's clearly not it since it would cause a very obvious deadlock (Frame.Abort calls ConnectionControl.End which locks _stateLock). So now I know what to do 😁

[santos222]

I'm getting HTTP 502.3 Bad Gateway (i have the correct Http Platform Handler installed). In the logs I see the following stack strace (towards the end I get the same error Unhandled Exception: Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -4082 EBUSY resource busy or locked ) Any ideas?

Hosting environment: Production
Now listening on: http://localhost:19685
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNet.Hosting.Internal.HostingEngine[1]
Request starting HTTP/1.1 GET http://localhost:60/api/values/5
info: Microsoft.AspNet.Mvc.Controllers.ControllerActionInvoker[1]
Executing action method Mvc6.Sample.Controllers.ValuesController.Get with arguments (5) - ModelState is Valid'
info: Microsoft.AspNet.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value Microsoft.AspNet.Mvc.ActionContext.
info: Microsoft.AspNet.Mvc.Infrastructure.MvcRouteHandler[2]
Executed action Mvc6.Sample.Controllers.ValuesController.Get in 0.0234ms
fail: Microsoft.AspNet.Server.Kestrel[0]
KestrelThread.DoPostWork
System.MissingMethodException: Method not found: 'Int32 System.Runtime.InteropServices.Marshal.SizeOf()'.
at Microsoft.AspNet.Server.Kestrel.Networking.UvWriteReq.Init(UvLoopHandle loop)
at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.WriteContext.DoWriteIfNeeded()
at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.WriteAllPending()
at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.<>c.b__15_0(SocketOutput _this)
at Microsoft.AspNet.Server.Kestrel.KestrelThread.<>c__301.<post>b__30_0(Object callback2, Object state2) at Microsoft.AspNet.Server.Kestrel.KestrelThread.DoPostWork() System.MissingMethodException: Method not found: 'Int32 System.Runtime.InteropServices.Marshal.SizeOf()'. at Microsoft.AspNet.Server.Kestrel.Networking.UvWriteReq.Init(UvLoopHandle loop) at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.WriteContext.DoWriteIfNeeded() at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.WriteAllPending() at Microsoft.AspNet.Server.Kestrel.Http.SocketOutput.<>c.<schedulewrite>b__15_0(SocketOutput _this) at Microsoft.AspNet.Server.Kestrel.KestrelThread.<>c__301.b__30_0(Object callback2, Object state2)
at Microsoft.AspNet.Server.Kestrel.KestrelThread.DoPostWork()
at Microsoft.AspNet.Server.Kestrel.KestrelThread.OnPost()
at Microsoft.AspNet.Server.Kestrel.Networking.UvAsyncHandle.AsyncCb(IntPtr handle)
at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.NativeMethods.uv_run(UvLoopHandle handle, Int32 mode)
at Microsoft.AspNet.Server.Kestre...(UvLoopHandle handle, Int32 mode)
at Microsoft.AspNet.Server.Kestrel.KestrelThread.ThreadStart(Object parameter)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.AspNet.Server.Kestrel.KestrelThread.Stop(TimeSpan timeout)
at Microsoft.AspNet.Server.Kestrel.KestrelEngine.Dispose()
at Microsoft.AspNet.Server.Kestrel.ServerFactory.<>c__DisplayClass4_0.b__0()
at Microsoft.AspNet.Server.Kestrel.Disposable.Dispose(Boolean disposing)
at Microsoft.AspNet.Server.Kestrel.Disposable.Dispose()
at Microsoft.AspNet.Hosting.Internal.HostingEngine.<>c__DisplayClass32_0.b__1()
at Microsoft.AspNet.Hosting.Internal.HostingEngine.Disposable.Dispose()
at Microsoft.AspNet.Hosting.Internal.Application.Dispose()
at Microsoft.AspNet.Hosting.WebApplication.Run(Type startupType, String[] args)
at Microsoft.AspNet.Server.Kestrel.Program.Main(String[] args)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Dnx.Runtime.Common.EntryPointExecutor.Execute(Assembly assembly, String[] args, IServiceProvider serviceProvider)
at Microsoft.Dnx.ApplicationHost.Program.<>c__DisplayClass3_0.b__0()
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
Unhandled Exception: Microsoft.AspNet.Server.Kestrel.Networking.UvException: Error -4082 EBUSY resource busy or locked
at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.Check(Int32 statusCode)
at Microsoft.AspNet.Server.Kestrel.Networking.Libuv.loop_close(UvLoopHandle handle)
at Microsoft.AspNet.Server.Kestrel.Networking.UvLoopHandle.ReleaseHandle()
at System.Runtime.InteropServices.SafeHandle.InternalFinalize()
at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
at System.Runtime.InteropServices.SafeHandle.Finalize()

[halter73]

@santos222 Your issue seems to have a different root cause:

System.MissingMethodException: Method not found: 'Int32 System.Runtime.InteropServices.Marshal.SizeOf()'

It could be an issue with mismatched dependencies. I would try deleting your ~/.nuget and APPLOCAL/NuGet directories and restore/build again.

[santos222]

@halter73 Same problem unfortunately, after performing those steps.

[cesarblum]

@santos222 Can you share more details about your setup? Are you using DNX or dotnet? Which version?

[cesarblum]

@santos222 Please open a new issue and share your details there. I'm closing this one since the change to fix it has been merged.

[santos222]

@CesarBS - Thanks, here it is: #694