netcore 3 intermittent http client errors

[stephenlautier]

After updating to netcore 3 we are getting several issues with the HttpClient, failing randomly - we have 2 scenarios which fails "similarly" in different projects.

Error 1

This is a running E2E test project performing a lot of HttpClient to call Api's and randomly a call fails, sometimes they do all work correctly.

	Exceptions: System.Net.Http.HttpRequestException: An error occurred while sending the request.
--
	 ---> System.IO.IOException: The response ended prematurely.
	   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
	   --- End of inner exception stack trace ---
	   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
	   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
	   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
	   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
	   at FluentlyHttpClient.Middleware.FluentMiddlewareHttpHandler.<>c__DisplayClass4_0.<<SendAsync>b__0>d.MoveNext()
	--- End of stack trace from previous location where exception was thrown ---
	   at FluentlyHttpClient.Middleware.ActionExecuteMiddleware.Invoke(FluentHttpMiddlewareContext context)
	   at FluentlyHttpClient.Middleware.TimerHttpMiddleware.Invoke(FluentHttpMiddlewareContext context)
	   at Tyr.Core.Middleware.LoggerFullHttpMiddleware.Invoke(FluentHttpMiddlewareContext context)
	   at Tyr.Core.Middleware.TyrErrorHttpMiddleware`1.Invoke(FluentHttpMiddlewareContext context)
	at Tyr.Core.Middleware.TyrErrorHttpMiddleware`1.Invoke(FluentHttpMiddlewareContext context)
	at FluentlyHttpClient.Middleware.FluentHttpMiddlewareRunner.Run(FluentHttpRequest request, Func`1 action)
	at FluentlyHttpClient.Middleware.FluentMiddlewareHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
	at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
	at FluentlyHttpClient.FluentHttpClient.Send(FluentHttpRequest request)
	at FluentlyHttpClient.FluentHttpRequestBuilder.ReturnAsResponse[T]()
	at Tyr.Midgard.Services.Auth.LoginCommand.Execute(TyrMidgardActionContext action)
	at Tyr.Midgard.Fixtures.Api.Auth.AuthMidgardFixture.A_login_action_is_requested(String username, String password)

Error 2

This is an aspnet project with SSR for angular - calls to node via aspnet node services, when adding a little bit of load they start failing randomly with this error.

An error occurred while sending the request.
--
	   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Microsoft.AspNetCore.NodeServices.HostingModels.HttpNodeInstance.InvokeExportAsync[T](NodeInvocationInfo invocationInfo, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.NodeServices.HostingModels.OutOfProcessNodeInstance.InvokeExportAsync[T](CancellationToken cancellationToken, String moduleName, String exportNameOrNull, Object[] args)
   at Microsoft.AspNetCore.NodeServices.NodeServicesImpl.InvokeExportWithPossibleRetryAsync[T](String moduleName, String exportedFunctionName, Object[] args, Boolean allowRetry, CancellationToken cancellationToken)
   at Midgard.Web.Controllers.HomeController.Index() in /app/Controllers/HomeController.cs:line 115
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

We are using

• Docker (linux) based on mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim
• netcore3

[scalablecory]

You're migrating to .NET Core 3.0 -- were you previously using .NET Core 2.x, or are you coming from .NET Framework 4.x?

"The response ended prematurely" indicates that HttpClient was expecting X bytes in a response and the connection was closed by the server before it could read all of them.

Are you able to reduce this behavior into a small repro?

[stephenlautier]

Sorry for not writing it, thought I did, We were netcore 2.x (specifically 2.2.1).

Unfortunately, the only 2 cases which we produce are work projects; Tried to produce one of them in a new project but didn't produce it, obviously our apps are much heavier than an empty project, and it seems when there are a lot in concurrency the failures increase.

Some extra info perhaps it helps...
In the second scenario we produce around 1mb HTML (which is quite big) from SSR (node) > dotnet and takes from 1.5sec - 6sec to fully complete - when i was "load" testing it, with 120 requests 40 concurrent was getting around ~4 failures (sometimes more) however when we decrease the concurrency to around 15 it was much more successful

[DaniilSokolyuk]

@stephenlautier
Same problem when rendering on nodejs but we use Jering.Javascript.NodeJS
I think problem in nodejs server

[stephenlautier]

@DaniilSokolyuk we get this even when we do not have any nodejs tho like the e2e scenario which i mentioned (error 1)

[karelz]

Triage: To make it actionable we need a small repro (without NodeJS server to avoid server problems).
Closing as it is not available and not actionable now. Feel free to reopen when you have the repro for us. Thanks!