SignalR .net client connects and disconnects immediately endlessly

[quahogrd]

Signalr .net client 2.2.1.0 on a windows server 2008 with .NET 4.0
Signalr server 2.2.1.0 on a windows 2012 server with .NET 4.5.

I start the client passing in the SSE as the client protocol and attach a ContinueWith() for exception handling. Sometimes the client goes into endless connect, immediate disconnect, reconnect, immediate disconnect loop.

[2016-11-18 18:44:44,505] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:44:44.5051561 - a135f3f8-2f7d-43d8-a3bb-87fa762337b4 - Connection Timeout Warning : Notifying user

[2016-11-18 18:44:50,027] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:44:50.0274499 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - OnConnectionSlow

[2016-11-18 18:44:53,896] [38] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:44:53.8961755 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - Connection Timed-out : Transport Lost Connection

[2016-11-18 18:45:01,711] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer:
23:45:01.7116252 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - ChangeState(Connected, Reconnecting)

[2016-11-18 18:45:05,564] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:45:05.5647511 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - SSE: GET https://xyz.com/signalr/signalr/reconnect?clientProtocol=1.4&transport=serverSentEvents&connectionData=[{"Name":"MyCompanyHub"}]&connectionToken=CYly4pzpvkfdo5VDApEcKWw4pmUfFfWwJ8bYjRtEesWzTRkCA4%2BIaXF%2FWzZNMBk3q%2FSew0KqQdnI4FZpmOHaZGhf20oFk4euY063WMOJO%2BAdZ9Mb5AXbSf2OckURI8eZq7PvjaO2ODg4ya7zeQyvu8UjHpujRZtAouoB6KJo4%3D&messageId=s-0%2C612EC3

[2016-11-18 18:45:25,579] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:45:25.5791662 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - ChangeState(Reconnecting, Connected)

[2016-11-18 18:45:30,680] [44] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:45:30.6802681 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - SSE: OnMessage(Data: initialized)

[2016-11-18 18:45:36,483] [38] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:45:36.4833565 - a135f328-2f7d-43b8-a3bb-87fa762337b4 - Connection Timed-out : Transport Lost Connection

While this is going on, the server is pushing data to the client without any issues. But the client is unable to respond back to the server and my UnobservedTaskException handler kicks in and logs:

System.InvalidOperationException: Connection started reconnecting before invocation result was received

From this point on, the client endlessly connects, immediately disconnects, reconnects, immediately disconnects, and so on.

[2016-11-18 18:47:40,064] [4] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:47:40.0641799 - f485d97f-8749-4446-bdb7-11b3d727e94e - SSE: GET https://xyz.com/signalr/signalr/connect?clientProtocol=1.4&transport=serverSentEvents&connectionData=[{"Name":"MyCompanyHub"}]&connectionToken=A9ANB8q1kC2Tns1Bb9y7GpO7T4zAaW%2FzP9AQcxVi4WvszZF8NjToy06jImrJBx0F8P6SRuIunGs5J701nRBG2aPN%2FAhm68eAyxYKeTw5yUSdiZWIe44TSAoOCjvmtMF7skgYT2oKjjKhhhj5YmxcJ8xWCD%2B2zlpJhxOa9loD0DQ%3D

[2016-11-18 18:47:53,261] [4] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:47:53.2615261 - f485d97f-8749-4446-bdb7-11b3d727e94e - Disconnected

[2016-11-18 18:49:02,243] [4] [(null)] DEBUG ExtensionMethods.TaskExtensionMethods - System.TimeoutException: Transport timed out trying to connect

[2016-11-18 18:47:57,083] [4] [(null)] DEBUG SignalRLogger- HubProxy.JsonSerializer: 23:47:57.0834526 - f485d97f-8749-4446-bdb7-11b3d727e94e - Transport.Dispose(f485d97f-8749-4446-bdb7-11b3d727e94e)

[2016-11-18 18:48:00,920] [4] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:48:00.9209788 - f485d97f-8749-4446-bdb7-11b3d727e94e - Closed

[2016-11-18 18:48:37,315] [4] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:48:37.3150789 - null - ChangeState(Disconnected, Connecting)

[2016-11-18 18:48:41,230] [3] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:48:41.2306036 - 375b4f0b-42d7-440c-9b00-146737eecc80 - SSE: GET https://xyz.com.com/signalr/signalr/connect?clientProtocol=1.4&transport=serverSentEvents&connectionData=[{"Name":"MyCompanyHub"}]&connectionToken=aPesIq1DlFL6oiqNJvXwOHGIQEP4cNufR5NJIjL0GmgGjaD3b4nvL7n7B%2Bl4Sb1caFr9ALQ3k8OEq0pAFrFyrQhe3VD%2FQ4cP6hvUgKAfhpV1tAf7tsRijHbmcEKtTYJiqFMEYgLSIq27BudjqSFia9HHbcGzXa42o9h%2BbsiGXRg%3D

[2016-11-18 18:48:54,568] [28] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:48:54.5683471 - 375b4f0b-42d7-440c-9b00-146737eecc80 - SSE: OnMessage(Data: initialized)

[2016-11-18 18:49:00,324] [3] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:49:00.3246364 - 375b4f0b-42d7-440c-9b00-146737eecc80 - Disconnected

[2016-11-18 18:49:02,243] [4] [(null)] DEBUG ExtensionMethods.TaskExtensionMethods - System.TimeoutException: Transport timed out trying to connect

[2016-11-18 18:49:07,984] [3] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:49:07.9840891 - 375b4f0b-42d7-440c-9b00-146737eecc80 - Transport.Dispose(375b4f0b-42d7-440c-9b00-146737eecc80)

[2016-11-18 18:49:15,643] [3] [(null)] DEBUG SignalRLogger - HubProxy.JsonSerializer: 23:49:15.6435418 - 375b4f0b-42d7-440c-9b00-146737eecc80 - Closed

The log above is continuously repeated until the client is restarted. Read some of the posts regarding this issues here and on StockOverFlow. All seem to suggest it was fixed in an earlier release than 2.2.1.0 but I am running into the same issue (I assume). Thanks

[moozzyk]

Once disconnected the client should not try to connect automatically. I guess you try to restart your client from the closed event. Can you show your code? Also, did you configure any timeouts on the server side?

[quahogrd]

Yes, I have a closed event handler - Retry() - which restarts the client.

private void Retry()
{
	//Cancellation is requested.
	if (_TokenSource.IsCancellationRequested)
		return;

	_Logger.Debug("SignalrClient.Retry()");

	lock (_Key)
	{
		if (_HubConnection.State != ConnectionState.Disconnected)
			return;

		//Make a http connection to the hub and loop until the http status code is 200.
		while (!SignalrService.CheckHubConnectionStatus())
		{
			SignalrService.Pause(1, 20, true);
		}
		StartClient();
	}   
}

private void StartClient()
{
	if (_TokenSource.IsCancellationRequested)
		return;

	_Logger.Debug("SignalrClient.StartClient()");

	var task1 = _HubConnection.Start(SignalrSettings.GetTransportProtocol())
		.ObserveException("SignalrClient.StartClient"); //Extension method with the ContinueWith()

	if (!task1.IsFaulted)
		  _Logger.Debug("SignalR client started...");
}

No, I haven't changed any timeouts. They are all default values.

So there is a HTTP connection to the hub while the client immediately disconnects after making the connection.

I changed the title of this posting to reflect the fact that the issue is the immediate disconnect after making the connection - not the reconnect itself as I am the one manually making it.

[moozzyk]

What do you do in ObserveException? How you are making sure that on this line if (!task1.IsFaulted) the task has been completed?

[quahogrd]

Here is the extension method.

public static Task ObserveException(this Task task, string taskName=null)
{
	return task.ContinueWith(ant =>
	{
		_Log.Debug("Observing the exception(s) thrown from the antecedent Task: " + taskName);
		ant.Exception?.Handle(x =>
		{
			_Log.Debug(x);
			return true;
		});

	}, TaskContinuationOptions.OnlyOnFaulted);
}

Earlier I had task1.Wait() in a try/catch but chose to go with the ContinueWith().

[moozzyk]

Is the trace from the time where you have task1.Wait()? Because task1.Wait() would be causing a deadlock as per #2852 and #2915. With ContinueWith on the other hand you are not waiting for the client to be connected so you will run into issues because you are using an object which is in a transient state. I suggest you use async/await all the way.

[quahogrd]

Unfortunately, I don't have the trace from when I had the task1.Wait() but I think this issue has started to happen right after using ContinueWith(). I didn't remember ever having this problem while using task1.Wait(). Also the client runs inside a windows service on a server with .NET 4.0. Asyn/await is available in 4.5. We have no control over the .NET version on the client side.

Are you suggesting the task1.IsFaulted is causing this problem as task1 could be in transient state at that point? How about if I pause (not using Thread.Sleep()) for sometime before checking the faulted status?

[moozzyk]

Because you don't wait for the task to finish your code continues to run while the connection is being started so you might be running code not intended to run when the connection is not connected.
With regards to .NET Framework 4 (as well as 4.5 and 4.5.1) they are no longer supported as per https://support.microsoft.com/en-us/help/17455/lifecycle-support-policy-faq-net-framework.