Exception at System.Net.Security.SslStreamInternal.EndRead when proxy request from Nginx to Kestrel through HTTPS

[yuwaMSFT]

I was trying out the SSL feature on Kestrel. When putting Nginx in front to proxy request to Kestrel server via https, I always got the following exception for each client request:

\funfile\Scratch\yuwa\Failures\041216\kestrelhttps\kestrellog.log

fail: Microsoft.AspNetCore.Server.Kestrel[0]
      FilteredStreamAdapter.CopyToAsync
System.AggregateException: One or more errors occurred. (The read operation failed, see inner exception.) ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.Threading.Tasks.TaskCanceledException: The request was aborted
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInput.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInputExtensions.<ReadAsyncAwaited>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Server.Kestrel.Filter.LibuvStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamAsyncHelper.EndRead(IAsyncResult asyncResult)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.EndRead(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<ReadAsync>b__85_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Filter.StreamExtensions.<CopyToAsync>d__0.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.IO.IOException: The read operation failed, see inner exception. ---> System.Threading.Tasks.TaskCanceledException: The request was aborted
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInput.GetResult()
   at Microsoft.AspNetCore.Server.Kestrel.Http.SocketInputExtensions.<ReadAsyncAwaited>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Microsoft.AspNetCore.Server.Kestrel.Filter.LibuvStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.IO.StreamAsyncHelper.EndRead(IAsyncResult asyncResult)
   at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.EndRead(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<ReadAsync>b__85_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Filter.StreamExtensions.<CopyToAsync>d__0.MoveNext()<---

When request from wget directly on local machine, I didn’t see such error. It seems something related to how Nginx terminates the https proxy connection. Is this a benign exception? If so, can we suppress it since it would fire for each https request proxied by Nginx.

BTW, I used the self-signed test cert (but I don’t think that’s the problem here).

[yuwaMSFT]

@halter73 @muratg @sivagms @sajayantony

[muratg]

@yuwaMSFT Looks like a server side error, right... Was the response fine on the client side?

[yuwaMSFT]

@muratg Yes it's server side error. Response to client seemed to be fine.

[cesarblum]

@yuwaMSFT What do you have in nginx.conf?

[yuwaMSFT]

This is the Nginx configuration:

            server {
                    listen 8081 ssl;
                    ssl_certificate     /etc/ssl/nginx/new.cert.cert;
                    ssl_certificate_key /etc/ssl/nginx/new.cert.key;
                    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;

                    location / {
                            proxy_set_header    Host $host;
                            proxy_set_header    X-Real-IP   $remote_addr;
                            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
                            access_log  off;
                            error_log on;
                            proxy_pass  https://localhost:5001;
                            proxy_set_header Upgrade $http_upgrade;
                            proxy_set_header Connection "upgrade";
                   }
            }

[yuwaMSFT]

BTW, this is the configuration for the site serving as the reverse proxy. nginx.conf itself was the default one after installing nginx.

[cesarblum]

What version of nginx are you using?

[yuwaMSFT]

nginx version: nginx/1.4.6 (Ubuntu)

[benaadams]

What happens if you add keep alive? Might be ngnix doing a close and the filter getting confused by it

[cesarblum]

@benaadams That's what I was about to try 😄

[benaadams]

There is also some oddness with ngnix, though not likely to be effecting this https://blog.cloudflare.com/the-curious-case-of-slow-downloads/

[yuwaMSFT]

client should already send out request with keep-alive.

Added the header for proxy in the site config:
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";

also set the ssl session cache in nginx.conf
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;

Even setting so, it seems nginx is still doing new connection for each https request from client. Probably it's because nginx needs to decrypt and re-encrypt for each request coming from client. Not sure if there is way to configure nginx to "cache" the proxy https connection...

If client hits Kestrel server directly without nginx in front, it seems the first request would hit this exception, but not subsequent requests, unless browser does refresh.

So this seems to be happening for each new HTTPS connection. Probably it's related to the HTTPS handshaking process?

[cesarblum]

@benaadams Great article, useful knowledge for us in Kestrel too 👍

[cesarblum]

It's interesting that it doesn't repro on Windows.

[cesarblum]

I can repro on Ubuntu with nginx 1.4.7, but it doesn't repro with nginx 1.8.1 (latest stable).

[cesarblum]

Looks like an nginx issue, I'm inclined to close this. @muratg thoughts?

[yuwaMSFT]

Even without nginx, I am seeing this exception for the first request from client browser.

[yuwaMSFT]

However curl from the local machine, I don't see any error.
But if using IE browser from another Windows machine against the same Kestrel server on Linux (without Nginx), I am seeing the same exception for the first request or on refresh.

[cesarblum]

Yeah, forget what I said earlier. It's not as consistent with the latest nginx, but after a few more requests I saw the exception again.

[benaadams]

corefx needs to implement ReadAsync and WriteAsync on SSLStream, I have no end of problems with the wrappers on BeginRead and EndRead #588

[cesarblum]

I didn't get to the bottom of it, but I was able to work around the problem by changing my nginx.conf from:

proxy_pass  https://localhost:5001;

to:

proxy_pass  http://localhost:5000;
proxy_redirect http://localhost:5000 https://localhost:5001;

with all else remaining the same.

[cesarblum]

It seems like nginx is doing something funny in the connection, but I don't know what exactly (I'm not well versed in tcpdump, trying to learn some to get to the bottom of this). From the logs I can see that we get a FIN from nginx, but for some reason the code is stuck at

KestrelHttpServer/src/Microsoft.AspNetCore.Server.Kestrel/Http/SocketInputExtensions.cs

Line 39 in 50f187a

await input;

although we do consume all input. When SocketInput is disposed later, it causes the last call to ReadAsync in

KestrelHttpServer/src/Microsoft.AspNetCore.Server.Kestrel/Filter/StreamExtensions.cs

Line 15 in f15471b

while ((bytesRead = await source.ReadAsync(block.Array, block.Data.Offset, block.Data.Count)) != 0)

to return a faulted task, with the error coming from

KestrelHttpServer/src/Microsoft.AspNetCore.Server.Kestrel/Http/SocketInput.cs

Line 209 in f15471b

_awaitableError = new TaskCanceledException("The request was aborted");

because Dispose() calls AbortAwaiting(). And all that leads to the exception seen in this issue.

[cesarblum]

@yuwaMSFT What browser did you use? I'm not reproing trying to hit from Chrome on another machine.

[yuwaMSFT]

@CesarBS I am seeing this on both Chrome and IE on a Windows client machine.

[yuwaMSFT]

@CesarBS Also, the server is a Ubuntu box in Azure, the client is on Corp net...