New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HttpClient GetAsync - The server returned an invalid or unrecognized response #17558

Closed
deepumi opened this Issue Mar 27, 2017 · 8 comments

Comments

Projects
None yet
4 participants
@deepumi

deepumi commented Mar 27, 2017

Version - .Net Core 1.0.1
OS - CENT OS.7 64 bit / Azure
Curl version

curl 7.53.1 (x86_64-pc-linux-gnu) libcurl/7.53.1 OpenSSL/1.0.1e zlib/1.2.7
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets HTTPS-proxy

When I make a request to http://www.pma.org/home/default.asp? URL I get the following error in my .Net Core App. However, If I make the request using Curl works fine.

System.Net.Http.HttpRequestException: The server returned an invalid or unrecognized response.
   at System.Net.Http.CurlResponseHeaderReader.ReadHeader(String& headerName, String& headerValue)
   at System.Net.Http.CurlHandler.MultiAgent.CurlReceiveHeadersCallback(IntPtr buffer, UInt64 size, UInt64 nitems, IntPtr context)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at hwapp.Program.<MakeWebRequest>d__1.MoveNext()

Sample code

try
{
   using (var client = new HttpClient())
   {
       client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
       client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36");
       var result = client.GetAsync("http://www.pma.org/home/default.asp?").GetAwaiter().GetResult();
       Console.WriteLine(result.StatusCode.ToString());
   }
}
catch (Exception exception)
{
   Console.WriteLine(exception.ToString());
}

Curl request

curl -I http://www.pma.org/home/default.asp?

Curl response

HTTP/1.1 200 OK
Date: Mon, 27 Mar 2017 17:30:18 GMT
Cache-Control no-cache:
Content-Length: 52904
Content-Type: text/html
Set-Cookie: BadIP=False; path=/
Set-Cookie: ASPSESSIONIDSSBBABDD=MKLKBPNDEHNNILAPDDBFGMML; path=/
Cache-control: private
X-FRAME-OPTIONS: SAMEORIGIN
Set-Cookie: BNES_BadIP=teIjVVdWBFXCMdVefZ8/VmOmIVKTmEJMKu74WIqHbXLL4Pb07YvoIqBkWdnZ+637; path=/
Set-Cookie: BNES_ASPSESSIONIDSSBBABDD=7twirxWeZB93d7p1vpjt0+EHI3recOI89u/w/4G0RaMDPdVQqDKsBJTzSFFHOynq22QKkNkv83C/5Le/uFbY2T5MTuFzFeVQkk1A6A+oXvg=; path=/

What's the problem?

@stephentoub

This comment has been minimized.

Show comment
Hide comment
@stephentoub

stephentoub Mar 27, 2017

Member

What's the problem?

The response header:

Cache-Control no-cache:

should be:

Cache-Control: no-cache
Member

stephentoub commented Mar 27, 2017

What's the problem?

The response header:

Cache-Control no-cache:

should be:

Cache-Control: no-cache
@stephentoub

This comment has been minimized.

Show comment
Hide comment
@stephentoub

stephentoub Mar 27, 2017

Member

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.
cc: @Priya91, @justinvp

Member

stephentoub commented Mar 27, 2017

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.
cc: @Priya91, @justinvp

@deepumi

This comment has been minimized.

Show comment
Hide comment
@deepumi

deepumi Mar 27, 2017

@stephentoub
Just now I tried adding Cache control, but no luck.

client.DefaultRequestHeaders.TryAddWithoutValidation("Cache-Control", "no-cache");

deepumi commented Mar 27, 2017

@stephentoub
Just now I tried adding Cache control, but no luck.

client.DefaultRequestHeaders.TryAddWithoutValidation("Cache-Control", "no-cache");

@davidsh

This comment has been minimized.

Show comment
Hide comment
@davidsh

davidsh Mar 27, 2017

Member

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.

I would be really careful about this. We need to constrain that type of loose parsing. I would definitely compare to what .NET Framework and perhaps other HTTP stacks do regarding this type of thing.

Member

davidsh commented Mar 27, 2017

It seems like we should make the response parser a bit less strict. This is technically an invalid response, but we could easily, for example, just ignore an invalid header of this kind rather than outright failing.

I would be really careful about this. We need to constrain that type of loose parsing. I would definitely compare to what .NET Framework and perhaps other HTTP stacks do regarding this type of thing.

@stephentoub

This comment has been minimized.

Show comment
Hide comment
@stephentoub

stephentoub Mar 27, 2017

Member

Just now I tried adding Cache control, but no luck.

Not in the request, in the response: this isn't about the code you're writing, rather the server is sending back an invalid response.

We need to constrain that type of loose parsing.

Ok. This example fails on all three of the .NET Framework, .NET Core on Windows, and .NET Core on Unix.

and perhaps other HTTP stacks do regarding this type of thing.

From the output shown in the original issue post, it looks like curl allows it through.

Member

stephentoub commented Mar 27, 2017

Just now I tried adding Cache control, but no luck.

Not in the request, in the response: this isn't about the code you're writing, rather the server is sending back an invalid response.

We need to constrain that type of loose parsing.

Ok. This example fails on all three of the .NET Framework, .NET Core on Windows, and .NET Core on Unix.

and perhaps other HTTP stacks do regarding this type of thing.

From the output shown in the original issue post, it looks like curl allows it through.

@deepumi

This comment has been minimized.

Show comment
Hide comment
@deepumi

deepumi Mar 27, 2017

@stephentoub Got it. Thank you.

From the output shown in the original issue post, it looks like curl allows it through.

Yes, CURL allow this and works fine.

deepumi commented Mar 27, 2017

@stephentoub Got it. Thank you.

From the output shown in the original issue post, it looks like curl allows it through.

Yes, CURL allow this and works fine.

@stephentoub stephentoub removed the os-linux label Mar 28, 2017

@stephentoub stephentoub added this to the 2.0.0 milestone Mar 28, 2017

@stephentoub

This comment has been minimized.

Show comment
Hide comment
@stephentoub

stephentoub Mar 29, 2017

Member

@davidsh, I removed the os-linux label as the behavior here is consistent across .NET Framework and .NET Core on both Windows and Linux. If you believe we should keep the strict parsing, please close. Thanks.

Member

stephentoub commented Mar 29, 2017

@davidsh, I removed the os-linux label as the behavior here is consistent across .NET Framework and .NET Core on both Windows and Linux. If you believe we should keep the strict parsing, please close. Thanks.

@davidsh

This comment has been minimized.

Show comment
Hide comment
@davidsh

davidsh Mar 29, 2017

Member

The server is violating the RFC by sending out a response header that is malformed.

The behavior is consistent between .NET Framework and .NET Core. And even if we wanted to change the behavior to be more loose in parsing, we can't change the OS WinHTTP behavior easily.

So, closing this issue as by-design.

Member

davidsh commented Mar 29, 2017

The server is violating the RFC by sending out a response header that is malformed.

The behavior is consistent between .NET Framework and .NET Core. And even if we wanted to change the behavior to be more loose in parsing, we can't change the OS WinHTTP behavior easily.

So, closing this issue as by-design.

@davidsh davidsh closed this Mar 29, 2017

@karelz karelz added the question label Mar 29, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment