HttpClient GetAsync - The server returned an invalid or unrecognized response

[deepumi]

Version - .Net Core 1.0.1
OS - CENT OS.7 64 bit / Azure
Curl version

curl 7.53.1 (x86_64-pc-linux-gnu) libcurl/7.53.1 OpenSSL/1.0.1e zlib/1.2.7
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets HTTPS-proxy

When I make a request to http://www.pma.org/home/default.asp? URL I get the following error in my .Net Core App. However, If I make the request using Curl works fine.

System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response.
   at System.Net.Http.CurlResponseHeaderReader.ReadHeader(String& headerName, String& headerValue)
   at System.Net.Http.CurlHandler.MultiAgent.CurlReceiveHeadersCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at hwapp.Program.<MakeWebRequest>d__1.MoveNext()

Sample code

try
{
   using (var client = new HttpClient())
   {
       client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
       client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");
       var result = client.GetAsync("http://www.pma.org/home/default.asp?").GetAwaiter().GetResult();
       Console.WriteLine(result.StatusCode.ToString());
   }
}
catch (Exception exception)
{
   Console.WriteLine(exception.ToString());
}

Curl request

curl -I http://www.pma.org/home/default.asp?

Curl response

HTTP/1.1 200 OK
Date: Mon, 27 Mar 2017 17:30:18 GMT
Cache-Control no-cache:
Content-Length: 52904
Content-Type: text/html
Set-Cookie: BadIP=False; path=/
Set-Cookie: ASPSESSIONIDSSBBABDD=MKLKBPNDEHNNILAPDDBFGMML; path=/
Cache-control: private
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: BNES_BadIP=teIjVVdWBFXCMdVefZ8/VmOmIVKTmEJMKu74WIqHbXLL4Pb07YvoIqBkWdnZ+637; path=/
Set-Cookie: BNES_ASPSESSIONIDSSBBABDD=7twirxWeZB93d7p1vpjt0+EHI3recOI89u/w/4G0RaMDPdVQqDKsBJTzSFFHOynq22QKkNkv83C/5Le/uFbY2T5MTuFzFeVQkk1A6A+oXvg=; path=/

What's the problem?

[stephentoub]

What's the problem?

The response header:

Cache-Control no-cache:

should be:

Cache-Control: no-cache

[stephentoub]

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.
cc: @Priya91, @justinvp

[deepumi]

@stephentoub
Just now I tried adding Cache control, but no luck.

client.DefaultRequestHeaders.TryAddWithoutValidation("Cache-Control", "no-cache");

[davidsh]

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.

I would be really careful about this. We need to constrain that type of loose parsing. I would definitely compare to what .NET Framework and perhaps other HTTP stacks do regarding this type of thing.

[stephentoub]

Just now I tried adding Cache control, but no luck.

Not in the request, in the response: this isn't about the code you're writing, rather the server is sending back an invalid response.

We need to constrain that type of loose parsing.

Ok. This example fails on all three of the .NET Framework, .NET Core on Windows, and .NET Core on Unix.

and perhaps other HTTP stacks do regarding this type of thing.

From the output shown in the original issue post, it looks like curl allows it through.

[deepumi]

@stephentoub Got it. Thank you.

From the output shown in the original issue post, it looks like curl allows it through.

Yes, CURL allow this and works fine.

[stephentoub]

@davidsh, I removed the os-linux label as the behavior here is consistent across .NET Framework and .NET Core on both Windows and Linux. If you believe we should keep the strict parsing, please close. Thanks.

[davidsh]

The server is violating the RFC by sending out a response header that is malformed.

The behavior is consistent between .NET Framework and .NET Core. And even if we wanted to change the behavior to be more loose in parsing, we can't change the OS WinHTTP behavior easily.

So, closing this issue as by-design.