[Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.

[yhan]

Hi,

on a HTTP POST, our kestrel server had the following problem.
I would like to know the whether the problem is due to OS slowness, the communication between OS and kestrel OR kestrel itself?

2018-08-27 08:25:11,172 [72] [ERROR] [ReadAPI.startup] - Subject=[Caraibes.Read.WebApi] Unhandled exception in Caraibes, Body=Unhandled exception occured. ContextId: '29790307-1a1f-45ad-a12c-1e250ab093c0'
Exception: [Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Toolbox.WebApi.Helpers.Middlewares.HttpRequestResponseLoggingMiddleware.d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Toolbox.WebApi.Helpers.Middlewares.HttpRequestResponseLoggingMiddleware.d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.d__6.MoveNext()], Detail=

[blowdart]

None of the above, it's data coming from the client that's under the minimum rate.

[yhan]

@blowdart Thanks for your prompt reply.
We have randomly this issue. As per your comment, in other words, it is a network issue? ( both the client and server are on the same LAN).
Beside retrying client side, are there any other neater ways to mitigate this problem?

[marfenij]

I also have such issue on GKE container 2.1.401 SDK; using LibUv transport - cause socket consume a lot CPU

Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
        at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync
        at System.IO.Pipelines.PipeCompletion.ThrowLatchedException
        at System.IO.Pipelines.Pipe.GetReadResult
        at System.IO.Pipelines.Pipe.GetReadAsyncResult
        at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal
        at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync
        at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync

or

Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Unexpected end of request content.
        at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync
        at System.IO.Pipelines.PipeCompletion.ThrowLatchedException
        at System.IO.Pipelines.Pipe.GetReadResult
        at System.IO.Pipelines.Pipe.GetReadAsyncResult
        at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync
        at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal
        at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.ReadAsync
        at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync

but kestrel return 200 OK

[blowdart]

You could lower the minimum request rate, but it sounds like the client isn't sending fast enough, I'd examine that first, if you're just sending a single, non-chunked request it should be fine, but there could be something weird going on in the network somewhere.

[marfenij]

I try that code in controller

                var minRequestRateFeature = this.HttpContext.Features.Get<IHttpMinRequestBodyDataRateFeature>();
                if (minRequestRateFeature != null)
                {
                    minRequestRateFeature.MinDataRate = new MinDataRate(100, TimeSpan.FromMinutes(5));
                }

                var minResponseRateFeature = this.HttpContext.Features.Get<IHttpMinResponseDataRateFeature>();
                if (minResponseRateFeature != null)
                {
                    minResponseRateFeature.MinDataRate = new MinDataRate(100, TimeSpan.FromMinutes(5));
                }

but it's not help

[blowdart]

@halter73

[halter73]

@marfenij This is most likely an issue with slowness in the client. I don't necessarily recommend doing this in production unless you restrict access to only trusted clients, but if you set the MinDataRates to null do you still see these issues?

[marfenij]

@halter73 thank you, I think that is client (mobile iOS) or network problem, but no idea how to get facts.
I'll try tomorrow,
also that method behind authorization

[muratg]

Closing this.

[wonea]

@marfenij Did you try this solution? I have similar situation of mobile clients interacting with the server.

[vmachacek]

This just started occurring on our server, and it's also from mobile traffic.

[halter73]

@vmachacek @wonea Have you tried decreasing the MinRequestBodyDataRate or increasing the graced period? If the issue is with slow/high-latency clients, that could help.

Another thing to watch out is threadpool starvation. That can sometimes lead to false-positives in the min data rate detection logic.