WebSockets selected as transport but send from server fails

[clarkee1066]

description

Using the simple chat sample app, taking the same machine into different internet cafes and home locations. Websockets is auto-selected as transport but send from server to client fails in some locations. An error is logged in transports.log when Clients.All.broadcastMessage() is called.

Logging on server reveals this error System.Net.WebSockets.WebSocketException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request at System.Web.WebSockets.WebSocketPipe.<>c__DisplayClass7.<ReadFragmentAsync>b__6(Int32 hrError, Int32 cbIO, Boolean fUtf8Encoded, Boolean fFinalFragment, Boolean fClose)

Client console shows message sent from client to server ok. However, server logs show send from server to client fails.

Note I know this sounds dubious, but we have only had this issue with clients in some locations in Dubai. Various locations in the UK are working fine. Is it possible that something on the network in some Dubai locations prevents send from server to client over websockets? If this is true then the transport selection routines need to check and fallback to ServerSentEvents or other.

functional impact

Some users never receive messages from the server. (The workaround is to use $.connection.hub.start({ transport: ['serverSentEvents', 'longPolling', 'foreverFrame'] }) on the client and everything is fine because WebSockets is never used)

Repro steps

1. Create the sample chat application using SignalR 2.1.1 (note version 2.0.3 also fails)
2. Use a browser that supports WebSockets, eg Safari
3. Find a location that fails - I could provide locations in Dubai where this fails, but you're not really going to go there are you, so I'll provide all the traces that I can, and can assist with more info if you need it
4. observe Client console (attached)
5. record server errors (attached)

Expected result

websockets should not be selected as the transport if send from server to client is blocked in the network

Actual result

Send from server to client fails.

Forcing the transport to use 'serverSentEvents', 'longPolling', or 'foreverFrame' fixes the issue, but ideally WebSockets should work as this is the best transport.

Further technical details

Client console when chat message sent

Server transports.log when Clients.All.broadcastMessage() fails

SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.TransportHeartBeat Verbose: 0 : KeepAlive(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.WebSocketTransport Error: 0 : OnError(2e265225-6314-43d1-933c-67dd089f1d6e, System.Net.WebSockets.WebSocketException (0x800703E3): The I/O operation has been aborted because of either a thread exit or an application request
   at System.Web.WebSockets.WebSocketPipe.<>c__DisplayClass7.<ReadFragmentAsync>b__6(Int32 hrError, Int32 cbIO, Boolean fUtf8Encoded, Boolean fFinalFragment, Boolean fClose)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.WebSockets.AspNetWebSocket.<DoWork>d__2d`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.WebSockets.AspNetWebSocket.<>c__DisplayClass1a.<<ReceiveAsyncImpl>b__18>d__1c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNet.SignalR.WebSockets.WebSocketMessageReader.<ReadMessageAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNet.SignalR.WebSockets.WebSocketHandler.<ProcessWebSocketRequestAsync>d__e.MoveNext())
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2e265225-6314-43d1-933c-67dd089f1d6e is dead
SignalR.Transports.TransportHeartBeat Information: 0 : Removing connection 2e265225-6314-43d1-933c-67dd089f1d6e
SignalR.Transports.WebSocketTransport Information: 0 : Abort(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.WebSocketTransport Information: 0 : End(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (2e265225-6314-43d1-933c-67dd089f1d6e)
SignalR.Transports.TransportHeartBeat Information: 0 : Dispose(). Closing all connections

[EnCey]

We have a similar issue where the error I/O operation has been aborted because of either a thread exit or an application request suddenly shows up for mysterious reasons.

The error occurs when testing locally, very frequently on my machine, my colleagues don't see the error as frequently. Same network, same hardware, same basic setup (though I may have other applications installed / running).

The SignalR server in our case is self-hosted.

Complete stack trace:

SignalR.Transports.WebSocketTransport Error: 0 : OnError(2834a90b-4fb9-42d5-bb44-0766f6b89b51, System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details.  
---> System.Net.HttpListenerException (0x80004005): The I/O operation has been aborted because of either a thread exit or an application request
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WebSocketHttpListenerDuplexStream.<ReadAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__47.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__1.MoveNext()
   at System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(String methodName, Exception exception, CancellationToken cancellationToken, Boolean aborted)
   at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.AspNet.SignalR.WebSockets.WebSocketMessageReader.<ReadMessageAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNet.SignalR.WebSockets.WebSocketHandler.<ProcessWebSocketRequestAsync>d__e.MoveNext())
SignalR.Transports.WebSocketTransport Information: 0 : CloseSocket(2834a90b-4fb9-42d5-bb44-0766f6b89b51)
SignalR.Transports.TransportHeartBeat Verbose: 0 : 2834a90b-4fb9-42d5-bb44-0766f6b89b51 is dead
SignalR.Transports.TransportHeartBeat Verbose: 0 : Connection 2834a90b-4fb9-42d5-bb44-0766f6b89b51 exists. Closing previous connection.
SignalR.Transports.WebSocketTransport Information: 0 : End(2834a90b-4fb9-42d5-bb44-0766f6b89b51)
SignalR.Transports.WebSocketTransport Verbose: 0 : Cancel(2834a90b-4fb9-42d5-bb44-0766f6b89b51)
SignalR.Transports.WebSocketTransport Verbose: 0 : DrainWrites(2834a90b-4fb9-42d5-bb44-0766f6b89b51)
SignalR.Transports.WebSocketTransport Information: 0 : CompleteRequest (2834a90b-4fb9-42d5-bb44-0766f6b89b51)

[sk1tt1sh]

Are there any updates for this?
This indicates the underlying issue.
https://connect.microsoft.com/VisualStudio/feedbackdetail/view/976851/server-websocket-closed-abruptly-the-i-o-operation-has-been-aborted-because-of-either-a-thread-exit-or-an-application-request

Also no activity for 2 months. Does this affect System.Web.Websockets as well, or is it isolated to the Microsoft.Websockets preview?

[clarkee1066]

We're still unable to use WebSockets in our production environment because of this issue which is a shame.

[sk1tt1sh]

I've implemented a workaround using System.Net.WebSockets//System.Web.WebSockets.

It's pretty ridiculous but every 15 seconds I send a keep alive to the server, and the server responds with a stay alive. I have been running the clients connected to a server instance in azure for about an hour now and not seeing any disconnections or i/o abort messages. It sucks that it burns bandwidth needlessly from something that shouldn't need ping/pong.

This is using an ASP net project with a clienthandler.ashx - I'm surprised a trivial workaround such as this is yielding positive results.

[EnCey]

We also get the error periodically and are unable to fix it. While it occurs more frequently when testing locally, we do get error reports from our production system, about once every day.

We also now believe that this might be the cause for the following error that is logged at the client side:

SignalR connection error occurred
No transport could be initialized successfully. Try specifying a different transport or none at all for auto initialization.

Up to now we thought this was caused by clients behind firewalls (we used an Azure port in the 10000+ range), but now we switched to port 443 and still monitor this error.

We do not use alternative transport methods and rely exclusively on WebSockets. The error is not caused by older browser versions as we detect those before attempting to initiate a connection. The behaviour seems to match that described by @clarkee1066, but since we do not use fallback protocols the connection error occurs.

[sk1tt1sh]

@EnCey I have been testing with about 40 clients in azure and have not had a disconnect since implementing a ping/pong from the client. 15 seconds was the first number I tried... potentially shorter interval may be necessary, and longer may work too. (I believe the KeepAliveInterval in System.Net.WebSockets is 30 seconds -- Idk if it actually works or is broken)

http://msdn.microsoft.com/en-us/library/system.net.websockets.clientwebsocketoptions.keepaliveinterval(v=vs.110).aspx

Every 15 seconds on the client send a keep alive to the server, and make the server respond with "StayAlive" or whatever.

setInterval(function () {ws.send("KeepAlive")}, 15000);
ws.onmessage = function (evt)
{if(evt.data === "StayAlive"){connected=true;}};

It's 18 bytes every 15 sec / client. At 1000 clients that 18KB/15sec (4.3MB/hr) - However being azure, it's 2.1MB/hr that is data out. This is not much. If You're concerned with that number send a shorter string.

I don't guarantee this will work, but hell it's worth a shot isn't it?

[EnCey]

@sk1tt1sh I'm not sure I understand, do you perform this in addition or instead of SignalR's KeepAlive feature?

At any rate, this was among the things we checked and the KeepAlives are being sent as they should.The majority of our clients also doesn't report any connection problems (nor do we receive any logs that would indicate them), the HttpListenerException occurs infrequently for yet unknown reasons. If it would be something fundamental like no KeepAlive packets being sent we would see the error much more frequently.

[sk1tt1sh]

@EnCey I do not actually use SignalR I use the bare System.Net.WebSockets & System.Web.WebSockets on asp or wcf service. The keep alive is visible in wireshark, but for some reason the session still ends up half closing. As an example set up a client that can send/receive on a test azure instance. Wait 3-5 min without sending any data across the line till you see that error, then send a message from the client. The server still gets it! However if the server replies the client will not receive it. However if data is transmitted on the socket in a typical message it will not disconnect, nor will this exception occur.

[EnCey]

Well, that's not the case for me. I've opened our web-application (hosted on azure) and didn't touch it for 20 minutes. No connection slow events, no reconnects, just regular keep alive packets.

The first request I sent after 20 minutes worked as expected, as did all subsequent ones.
In my case the issue(s) occur sporadically for yet unknown reasons.

[chronicbint]

This happens to me when installed on the production web server. Do not see the error when running on my local iis.

[Necroogre]

Need install webSocket on server iis
https://docs.microsoft.com/en-us/iis/configuration/system.webServer/webSocket

[aspnet-hello]

This issue has been closed as part of issue clean-up as described in https://blogs.msdn.microsoft.com/webdev/2018/09/17/the-future-of-asp-net-signalr/. If you're still encountering this problem, please feel free to re-open and comment to let us know! We're still interested in hearing from you, the backlog just got a little big and we had to do a bulk clean up to get back on top of things. Thanks for your continued feedback!