Crash under high load

[ayende]

We have a reproducible crash on Kestrel when running a load test using Kestrel 1.1.0.

 ./wrk -d10s -t 32 -c 512 http://127.0.0.1:8080 -s batch-writes.lua -- 128 128

This generates 128 pipelined requests or about 30 KB each, on a 10 GiB network.

Unhandled Exception: System.ObjectDisposedException: Safe handle has been closed
   at System.Runtime.InteropServices.SafeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolean& success)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.NativeMethods.uv_read_stop(UvStreamHandle handle)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.read_stop(UvStreamHandle handle)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.UvStreamHandle.ReadStop()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.DoPostWork()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.OnPost()
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.NativeMethods.uv_run(UvLoopHandle handle, Int32 mode)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.Networking.Libuv.run(UvLoopHandle handle, Int32 mode)
   at Microsoft.AspNetCore.Server.Kestrel.Internal.KestrelThread.ThreadStart(Object parameter)

[davidfowl]

Can you share the lua script?

[ayende]

Sure, here:
https://github.com/ravendb/ravendb/blob/v4.0/bench/Scripts/batch-writes.lua

Note that this is running on Ubutnu

[davidfowl]

which version of ubuntu?

[davidfowl]

Also @ayende Is your application consuming the body?

[ayende]

Pretty sure 16.04, but I'll need to double check on the office.

And yes, we read the entire body (at least, we should be doing that).
It is possible that in this case requests are hanging on the queue long enough to become abandoned, I guess, but correctly processed request will read the entire buffer.

[aviviadi]

It happens on updated ubuntu 16.04 with dotnet core preview2.1-003177 .

[halter73]

@ayende If you could provide the project you're running wrk against to repro this issue that would be great. If it's small enough you could put it into a gist.

[Eilon]

@halter73 @davidfowl let's take this offline to review.

[Eilon]

@ayende if you could send the requested info directly to me at elipton@microsoft.com that would be preferable. Thanks!

[ayende]

Hi,
I'll try doing a small repro tomorrow.

It isn't small, but the project itself lives here:

https://github.com/ravendb/ravendb

With the v4.0 branch, you can generate the issue by getting it, doing dotnet restore, and then dotnet run -c Release src/Raven.Server.

That will run the server, you'll then need to create a database using:

curl 'http://localhost:8080/admin/databases?name=BenchmarkDB' -X PUT  --data-binary '{"Id":"BenchmarkDB","Settings":{"Raven/DataDir":"~/BenchmarkDB"},"SecuredSettings":{},"Disabled":false}' 

And then run the wrk command:

 ./wrk -d10s -t 32 -c 512 http://127.0.0.1:8080 -s batch-writes.lua -- 128 128

The lua script is:
https://github.com/ravendb/ravendb/blob/v4.0/bench/Scripts/batch-writes.lua

[Eilon]

@halter73 - Done?