Server unexpectedly returns http status 200 (OK) with empty content in release 2.1.0

[baal2000]

@halter73
After upgrading from AspNetCore NuGet package 2.1.0-rc1-final to 2.1.0-rtm-30787 that differ slightly with #2533 and #2562 we noticed instances of 200 server responses containing empty content. This behavior messes with JSON-RPC client functionality. The behavior is intermittent in our production environment. We can not reliably reproduce (i.e. trigger in a controllable manner).

In our view the change to optional _streams?.Abort in aspnet/KestrelHttpServer@6e003d8

            if (error != null)
            {
                 _streams?.Abort(error);	             
            }

could have had wider effect than for WebSocket connections only.

The issue #2530 mentions #2533 may or may not be similar and we are not sure about the timing wither it preceded or followed the change.

Note that HttpProtocol.Abort in dev branch has remained untouched v.s. release/2.1 leaving _streams?.Abort non-optional. It has been changed in #2612 but essentially remained intact.

Is this (the empty response symptom and the differences between the branches) something that you can give some insight or there is a duplicate issue I missed?

Thanks.

[karelz]

@davidfowl @muratg can you please help with this issue? This is part of larger .NET Core deployment which hit some issues back in December and we are slowly working through issues (after migrating to 2.1). Thanks!

[baal2000]

@karelz @halter73 @davidfowl
Interesting: disabling the transfer rate - based KestrelServerLimits and relaxing others on 2 test servers decreased the rate of empty responses generated by these servers 30x-fold (200 down to 7 instances per hour).

[muratg]

cc @Tratcher

[Tratcher]

Can you clarify "empty content"? Do the headers give a Content-Length or Transfer-Encoding:chunked expecting content but the body is truncated? Or do the headers not indicate any body is expected?

The connection is closed, correct?

[Tratcher]

What specific limits did you disable or relax? What were the new values?

[baal2000]

@Tratcher

• Not sure about the connection closed or not.
• Empty content: please see below stringResponse against >> is being an empty string:

            using (var httpRequestMessage = new HttpRequestMessage { Method = new HttpMethod("POST"),  RequestUri = new Uri(requestUrl) })
            {
                httpRequestMessage.Content = new StringContent(ToJSON(jsonRequest));
                using (var response = await httpClient.SendAsync(httpRequestMessage, cancellationToken).ConfigureAwait(false))
                {
                    if (!IsSuccessCode(response.StatusCode))
                    {
                        throw ....
                    }
>>                  var stringResponse = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
                    ...
                }
             }

• We do not request / expect chunked content here, do not set headers explicitly
• Not sure about response.Content.Headers.ContentLength value. Do you need one?
• Specific limits:
  *before: all defaults
  *after:

                // >>> Disable following limits:
                limits.MinRequestBodyDataRate = null;
                limits.MinResponseDataRate = null;
                // <<<

                // >>> 10 times default values.
                limits.KeepAliveTimeout = TimeSpan.FromMinutes(20); //2m
                limits.RequestHeadersTimeout = TimeSpan.FromSeconds(300);
                limits.MaxRequestBodySize = 10 * 30 * 1024 * 1024; // 300 Mb
                limits.MaxRequestBufferSize = 10 * 1024 * 1024; // 10Mb
                limits.MaxRequestLineSize = 10 * 8 * 1024; // 80 Kb
                limits.MaxRequestHeadersTotalSize = 10 * 32 * 1024; // 320 Kb
                limits.MaxRequestHeaderCount = 10 * 1000; 
                // <<<

Thanks!

[Tratcher]

That's not a good sign, SendAsync should have thrown for a truncated response. There's definitely a problem on the server side then. Are you using Kestrel with IIS or accessing it directly? The last issue I saw like this was in our IIS component (ANCM).

Any chance of doing some A/B testing to narrow down which limit settings affect your symptoms? E.g. I wouldn't expect the size or count settings to be related, only the rate or timeouts.

[baal2000]

@Tratcher
Sorry missed the last update.

We are using Kestrel directly.

We may try the type of test you asked if necessary, in time. Can you provide with the options to test?

Whether RequestHeadersTimeout, MinRequestBodyDataRate or MinResponseDataRate are being triggered: wouldn't either lead to the similar outcome on the server side?

Have you looked at the changes between 2.1.0-rc1-final and 2.1.0-rtm-30787per original post to see if there is some clue?

Thanks!

[Tratcher]

As I don't have a repro I'm trying to narrow down the scenario before digging too deeply into the code. Any testing you can do in parallel will help. MinRequestBodyDataRate and MinResponseDataRate are the most likely candidates.

[Tratcher]

How large is your JSON POST on average? And what about the response?

Also, are these failures on http or https requests?

[baal2000]

@Tratcher
https - only JSON POST is tiny. Looks like (parameter names changed):

JSON-RPC: {"method":"Authenticate","params":{"param1":"AAA","param2":"BBB","param1":"CCC"}}

The important part is that the client that received the invalid result makes 500 or so parallel async tasks calling to 500 individual servers (a server per call) and then waits for the individual tasks completion, checks the result, waits another etc. I guess the client may become very well overloaded with making these requests, some could become blocked temporarily due to the lack of available system resources.

[baal2000]

... And the response is similarly short.

[halter73]

@baal2000 Would it be possible for you to provide the complete failing HTTP response with empty content including headers? Thanks.

[baal2000]

@halter73
I'll try, it may take time to catch and record the response in full.