Need help. AsyncWampConnection may stuck at InnerSend while WebSocket is already closed?

[bigbearzhu]

I'm seeing memory leakage with lots of WampMessages stuck on a host Owin WebSocket connection. Could you see if there might be a bug in WampSharp?

I have attached some screenshot for reference.

1. 2.5GB memory held by AsyncWampConnection.InnerSend
   

2.WebSocketContext on OwinWebScoketWrapper already has disposed cancellation token and the ClientCloseStatus is 1000 (normal closure).

What I have noted is that the public CancellationToken property on OwinWebSocketWrapper is not used anywhere in the code. Not too sure whether that is as designed or not. Another thing is RaiseConnectionClosed is only on WebSocketWrapperConnection.RunAsync when there is any exception caught or a valid cancellation frame is received. Could it miss a scenario that the connection is closed with no cancellation frame right before while (mWebSocket.IsConnected) is called in WebSocketWrapperConnection.RunAsync? Should SendAsync also check if a connection is still alive?

[darkl]

Regarding your questions:

• I am not sure what is the purpose of the CancellationToken property. See its description in the OWIN spec (websocket.CallCancelled property).
• I don't know if there could be missing a scenario. I was assuming that either an exception will occur or a cancelation frame will be received.
• We already check in AsyncWampConnection.InnerSend whether the connection is alive or not, so adding a condition to SendAsync won't help. We can add this condition to AsyncWampConnection.Send but I'm not sure it will help much.

Elad

[darkl]

Sorry for closing the issue and the previous comment, the "Close and comment" button was accidentally hit.

[bigbearzhu]

I found something might be related to this. But not 100% sure.

We are using Owin WebSocket and we often see OperationCancelledException at WebSocketWrapperConnection.cs line 115 when client has closed a wamp connection normally.

The issue is that WebSocketWrapperConnection.cs line 113 trigger RaiseConnectionClosed event before it calls mWebSocket.CloseAsync. When RaiseConnectionClosed is triggered, it actually went through WampListener that disposes the WebSocketWrapperConnection asyncronously. Aparrently WebSocketWrapperConnection.Dispose cancels the mCancellationTokenSource. The stacktrace is as below:

WampSharp.Core.Listener.AsyncWampConnection<Newtonsoft.Json.Linq.JToken>.SystemEx-IAsyncDisposable-DisposeAsync Line 139	C#
WampSharp.Core.Listener.WampListener<Newtonsoft.Json.Linq.JToken,WampSharp.V2.Core.Contracts.IWampClientProxy<Newtonsoft.Json.Linq.JToken>>.OnCloseConnection Line 91	C#
WampSharp.V2.Core.Listener.WampListener<Newtonsoft.Json.Linq.JToken>.OnCloseConnection Line 63	C#
WampSharp.Core.Listener.WampListener<Newtonsoft.Json.Linq.JToken,WampSharp.V2.Core.Contracts.IWampClientProxy<Newtonsoft.Json.Linq.JToken>>.OnConnectionClosed Line 180	C#
WampSharp.Core.Listener.WampListener<Newtonsoft.Json.Linq.JToken,WampSharp.V2.Core.Contracts.IWampClientProxy<Newtonsoft.Json.Linq.JToken>>.OnConnectionClose Line 171	C#
WampSharp.Core.Listener.AsyncWampConnection<Newtonsoft.Json.Linq.JToken>.RaiseConnectionClosed Line 116	C#
WampSharp.WebSockets.WebSocketWrapperConnection<Newtonsoft.Json.Linq.JToken>.RunAsync Line 113	C#

So when mWebSocket.CloseAsync is called, the mCancellationTokenSource may have already been cancelled. And it triggers that OperationCancelledException. I don't know if CloseAsync would do everything correctly in that scenario. But I think line 113 and line 115 should be swapped to make sure RaiseConnectionClosed is fired only after the connection is really closed.

I have tried to reproduce this leak, doesn't seem able to though. However I think at least we should fix this. Thanks

[bigbearzhu]

From this code I guess the CallCancelled is only cancelled when there is any exception thrown in WampSharp RunAsync. But it is just disposed when RunAsync completes. I think it is almost doing the same thing as the mCancellationTokenSource in WebSocketWrapperConnection because RunAsync catch exceptions and cancel the token source.

My only concern is if RunAsync didn't receive Close frame for some reason, but there was no exception caught (e.g. connection becomes disconnected after received a full message), then RunAsync wouldn't trigger RasieConnectionClosed event. Then the mCancellationTokenSource wouldn't be canceled hence all pending SendAsync tasks. If you look at my memory snapshot, you will see that the CallCancelled was disposed but not cancelled. That means RunAsync probably finished with no error, but the mCancellationTokenSource is not disposed or cancelled. It matches the scenario above.

I'm thinking we probably should move RaiseConnectionClosed(); out from the while loop in RunSync so it is guaranteed to be closed?