HttpClient fails with "The server returned an invalid or unrecognized response" when using Apache mod_proxy

[bruno-garcia]

I have an issue in production where the heartbeat GetAsync requests that a Windows Server 2012 R2 sends using HttpClient fail intermittently. Around 1 in 10 requests fail with:

Unhandled Exception: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Program.Main(String[] args) 

The servers responding to these heartbeat are CentOS 7.1 with Apache mod_proxy version: httpd-2.4.6-45.el7.centos.4.x86_64
All applications are .NET Core 1.1.

I was able to narrow down the issue to the combination of Client on Windows and Kestrel behind Apache. I created a repo with a script to reproduce the issue.

I'm unable to reproduce the issue when the client is running on Linux.
Also, if I remove the break between requests, the issue doesn't not happen. If I change the interval from 5 seconds to 4 or 6 seconds, the issue doesn't happen either.

That is in fact my current work around.

[davidsh]

Are doing GET or POST requests? Do you have any traces?

Are you using 'Transfer-Encoding: chunked' requests instead of 'Content-Length: " semantics?

We did recently fix a bug in HttpClient/WinHttpHandler on Windows where chunked requests were being sent incorrectly and sometimes this resulted in getting back "The server returned an invalid or unrecognized response" errors (#19251). Not sure if it is related to your problem.

[bruno-garcia]

Hi @davidsh! It's a plain GetAsync. Not using 'Transfer-Encoding chuncked'. Not explicitly anyway.

My production scenario is more complex but I was able to reproduce it with such server:

app.Run(context =>
{
    context.Response.StatusCode = 204;
    return Task.CompletedTask;
});

And client simply:

var client = new HttpClient();
while (true)
{
    Console.WriteLine(client.GetAsync("server").GetAwaiter().GetResult().StatusCode);
    Thread.Sleep(5000);
}

Thanks!

[davidsh]

Can you repro this without using Kestrel? I.e. using IIS or another server?

There was a bug in Kestrel fixed by the ASP.NET team several months ago that caused it to send out invalid responses (extra chunked terminators) which causes problems with RFC strict HTTP stacks like HttpClient.

I don't remember the issue number but it did turn out to be a bug in Kestrel.

[davidsh]

Also, please attach a complete Visual Studio solution (client and server) in order to validate a repro. It the problem repro's easily then we can get traces and confirm the problem is Kestrel sending back invalid responses.

[bruno-garcia]

I can't reproduce it when HttpClient connects to Kestrel directly.
Only when there's apache mod_proxy in between.

The complete solution on how to reproduce it is here

It includes the Apache configuration.

Thanks again!

[bruno-garcia]

Whatever it is that the server writes to the response that HttpClient can't handle, it is not a problem to Fiddler4.
I can reproduce the issue constantly but as soon as I fire up fiddler4, the clients stop crashing.

[davidsh]

I can't reproduce it when HttpClient connects to Kestrel directly.
Only when there's apache mod_proxy in between.

This tells me that it is likely that the mod_proxy is transmitting invalid RFC data to the client.

[bruno-garcia]

Yes, indeed. But only fails when the client is running on Windows.
Running on Linux works just fine.

Also, I pointed apache to another site on the backend (instead of the kestrel server) and also could not reproduce it.

Seems to be a HttpClient on Windows not handling the response generated by Apache + kestrel.

The setup I have is what's described on official ASP.NET Core docs.

[davidsh]

It is possible that the "The server returned an invalid or unrecognized response" error is a limitation/bug in the WinHttpHandler HTTP stack (which uses native WinHTTP) on Windows.

Starting with .NET Core 2.1 (available now as Preview 2), there is a new HTTP stack, SocketsHttpHandler, that works on both Windows and Linux.

See:
https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/

If you are still seeing a problem with using the apache mod_proxy component, please try out .NET Core 2.1. If you are still having problems, please open up a new issue. Thanks!

cc: @karelz

[jugg1es]

Is there any work around for .net 2.0?