The server returned an invalid or unrecognized response" error when using System.Net.Http.HttpClient

[Petermarcu]

Getting this bug into the right place: dotnet/core#321

@corinas
Hello,

I am facing a problem when making concurrent requests to a web api when using dotnet core. Sometimes i'm getting the below error; the error is not happening all the time (usually it happens once or twice in 600 concurrent calls).

System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

Here's the test method I use for testing:

    [Fact]
    public void PostToStardogParallelTaskWithSendAsync()
    {
        int x = 600;
        Task[] allTasks = new Task[x];

        using (var httpClient = new HttpClient())
        {
            var byteArray = Encoding.ASCII.GetBytes($"user:password");                

            for (int k = 0; k < 1; k++)
            {
                for (int i = 0; i < x; i++)
                {
                    System.Diagnostics.Debug.WriteLine($"Starting tsk {i}...");
                    allTasks[i] = Task.Factory.StartNew(async (ii) =>
                    {
                        var d = DateTime.Now;
                        var query = $"INSERT DATA {{ <urn:subj> <urn:pred> <urn:obj>. }}";
                        var content = new StringContent(query, Encoding.UTF8, "application/sparql-update");

                        HttpRequestMessage rm = new HttpRequestMessage(HttpMethod.Post, "MyStardogWebApiUrl");
                        rm.Headers.TryAddWithoutValidation("accept", "text/boolean");
                        rm.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
                        rm.Content = content;                            
                        try
                        {
                            var response = await httpClient.SendAsync(rm);
                        }
                        catch (Exception e)
                        {
                            System.Diagnostics.Debug.WriteLine($"\nThe following exception have been thrown: {e}");                            
                        }

                        System.Diagnostics.Debug.WriteLine($"Task {ii} ended in: {(DateTime.Now - d).TotalMilliseconds}.");

                    }, i).Unwrap();
                }
                try
                {
                    Task.WaitAll(allTasks);
                }
                catch (Exception e)
                {
                    Console.WriteLine($"nThe following exception have been thrown: {e}");                       
                }
            }
        }
    }

When this error occurs, the response looks like this:
{StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: , Headers:{}}

Calling the same method from a testing project that uses dotnet framework 4.5 does not cause any issue.

Could you please advise?
Thanks!

Corina

[davidsh]

cc: @karelz

The repro code posted above is a just a piece of the total repro. What server is this talking to? In order to repro the problem, we need a complete solution. The error being reported is about parsing the server response. So, it is important to use the same server to debug this issue.

[karelz]

@corinas can you please provide more details here? Thanks!

[JosVerburg]

We've also encountered this issue. We only seem to get the unrecognized response when making HEAD requests to a server which serves a redirect.

System.AggregateException: One or more errors occurred. (An error occurred while sending the request.) 
---> System.Net.Http.HttpRequestException: An error occurred while sending the request. 
---> System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at ConsoleApp3.Program.Main() in d:\projects\ConsoleApp3\ConsoleApp3\Program.cs:line 23

---> (Inner Exception #0) System.Net.Http.HttpRequestException: An error occurred while sending the request. 
---> System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.WinHttpHandler.<StartRequest>d__105.MoveNext()
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()<---

We use WebListener from "Microsoft.AspNetCore.Server.WebListener": "1.1.1",.
For the purpose of reproducing the we've created the following simple web application:
Program.cs

namespace WebApplication1
{
	public class Program
	{
		public static void Main(string[] args)
		{
			var host = new WebHostBuilder()
				.UseWebListener()
				.UseContentRoot(Directory.GetCurrentDirectory())
				.UseUrls("http://test.com:5000")
				.UseStartup<Startup>()
				.Build();

			host.Run();
		}
	}
}

Startup.cs

namespace WebApplication1
{
	public class Startup
	{
		public void Configure(IApplicationBuilder app)
		{
			app.Run(context =>
			{
				if (context.Request.GetDisplayUrl().EndsWith("redirect.html"))
				{
					context.Response.StatusCode = 200;
					return Task.CompletedTask;
				}

				context.Response.Redirect("/redirect.html");

				return Task.CompletedTask;
			});
		}
	}
}

And created a console application which makes the requests.
Program.cs

namespace ConsoleApp3
{
	class Program
	{
		static void Main()
		{
			for (var i = 0; i < 10000; i++)
			{
				using (var httpClient = new HttpClient())
				{
					var request = new HttpRequestMessage(HttpMethod.Head, new Uri("http://test.com:5000"));
					var response = httpClient.SendAsync(request).Result;
				}
			}
		}
	}
}

[karelz]

@JosVerburg how is your repro and callstack related to the original one here?
If it is not confirmed it is dupe, please file a new issue to keep things separate, thank you!

[davidsh]

@JosVerburg Please put together a complete, simple, repro with a Visual Studio 2015 or 2017 solution. You can simply zip up the solution and post it here. That would be the best way for us to investigate.

[davidsh]

And the reason for a complete repro is that I tried your sample code above and was unable to put together a complete solution that would launch WebListener. Latest version of ASP.NET Core uses a different API for the Startup.Configure:

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)

[JosVerburg]

@davidsh I've setup a repo with the solution at https://github.com/JosVerburg/ReproduceUnrecognizedResponse.

@karelz To me it seems like the same issue, if you still can't reproduce it with the example to confirm I'll make a separate issue.

Additional information:
We've also inspected the response and request using fiddler
Request:

HEAD http://test.com:5000/ HTTP/1.1
Connection: Keep-Alive
Accept-Encoding: gzip, deflate
Host: test.com:5000

Response:

HTTP/1.1 302 Found
Transfer-Encoding: chunked
Location: /redirect.html
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 30 Mar 2017 07:45:19 GMT

0

[karelz]

@JosVerburg why do you think it is the same issue? What is similar? (the callstack looks different to me)

[karelz]

OK, I updated the callstacks in top post and latest reply and now I see the similarities. he inner exception in interesting in your case. Sorry for the confusion ...

[davidsh]

@JosVerburg

Thanks for posting a complete repro!

I build and ran the solution. However, I got the following exception:

Your client code is incorrect. HttpClient design only allows an HttpRequestMessage to be sent ONCE. It's lifetime is over after one request. Due to this exception, I was unable to repro the "The server returned an invalid or unrecognized response" exception.

Can you revise your repro so that it doesn't try to reuse the same HttpRequestMessage? Perhaps that is the actual problem?

[davidsh]

Actually, after looking at your code again, you are creating a new HttpRequestMessage each time! So, this is very strange because that exception should not be returned.

I ran the repro again and was able to generate the exception we're investigating.

The exception happened after 906 iterations.

We will investigate. Thx.

[devatwork]

I've seen this issue consistently on Windows Server 2012 R2 running on .net core 1.0.

I'm using ASP.NET Core Proxy to proxy a number of requests to another application running on WebListener. Proxy internally uses HttpClient.

From the looks of it (applications set up and stacktrack), it the same issue as reported by @JosVerburg.

Is there anything I can contribute in order to get a diagnosis of this issue?

[JosVerburg]

We've also tried to reproduce the issue with wget.
wget did not indicate an invalid response.

Fiddler also doesn't indicate an invalid response.

[devatwork]

I've written a test client for full framework, see https://github.com/devatwork/ReproduceUnrecognizedResponse. Interestingly enough full framework also thinks this is an issue on the server side as well. Stack trace:

System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

This seems to suggest that the server application returns an invalid response. In both Jos's case and my case the server application is using WebListener/HttpSysServer. The HTTP response messages are identical according to a Fiddler trace I've run.