What is the proper way to reconnect?

[alexiordan]

Since reconnect is gone, when using WebSocket as transport layer what is the advisable way for attempting to maintain a connection always on (by attempting reconnection when connection drop is detected)?

It is more like a documentation issue since I didn't find it in documentation.

On server side I have set a KeepAlive of 3 seconds. I have also seen a NegotiateTimeout property. What is it for?

services.AddSignalR(
                options =>
                {
                    options.KeepAliveInterval = TimeSpan.FromSeconds(3);
                    //What is this for?
                    //options.NegotiateTimeout
                });

On client side I have requested for WebSockets with KeepAlive 3 seconds.

 HubConnection = new HubConnectionBuilder()
                    .WithTransport(Microsoft.AspNetCore.Sockets.TransportType.WebSockets)
                    .WithWebSocketOptions(a => a.KeepAliveInterval = TimeSpan.FromSeconds(3))
                    .Build();

There is also ServerTimeout option. What is it for?

HubConnection.ServerTimeout = TimeSpan.FromSeconds(9);

I am currently handling the closing event and performing following actions:

await HubConnection.StopAsync();
await Task.Delay(2000);
 HubConnection = new HubConnectionBuilder()
                    .WithTransport(Microsoft.AspNetCore.Sockets.TransportType.WebSockets)
                    .WithWebSocketOptions(a => a.KeepAliveInterval = TimeSpan.FromSeconds(3))
                    .Build();
//subscribe to events
await HubConnection.StartAsync();

Is this the correct way to do it, by building a new HubConnection on closing?

Before StopAsync() and starting it again I placed a 2 seconds delay. Should I keep it?

[analogrelay]

I have also seen a NegotiateTimeout property.

After establishing the connection, the client is expected to send a message indicating the protocol it wishes to use. The NegotiateTimeout is how long the server will wait for that message to be sent. The default value should be totally fine for most, if not all, use cases.

There is also ServerTimeout option. What is it for?

If the client hasn't heard from the server within the ServerTimeout, it will close the connection and report an error indicating that the server is not present. This should always be significantly larger than the KeepAliveInterval on the server. The KeepAliveInterval on the server configures the server to send a keep-alive message within that interval if no other messages have been sent.

I am currently handling the closing event and performing following actions:

You actually don't have to re-create the HubConnection. You can simply call StartAsync on it again. The reason this flow is manual is that it is very likely you will want to write application logic for the interval between disconnect and reconnect, especially since SignalR no longer buffers messages for you. You could display a message to a user or disable some UI, for example.

Before StopAsync() and starting it again I placed a 2 seconds delay. Should I keep it?

This depends a lot on what you see in the wild. For example, if the server has 1000 clients connected and it goes down, when it comes back, it'll have 1000 clients immediately trying to reconnect (this was known as a "Reconnect Storm" in SignalR for ASP.NET). You should be prepared for this scenario and see if you can put in some diagnostics to understand client behavior when this happens. You could have clients pick a random delay between when the disconnect occurs and when reconnection happens to help mitigate this.

[alexiordan]

OK, now if I don't re-create the HubConnection I get the following behavior:
On Closed Event after some "A connection with the server could not be established" exceptions I get this:

Exception thrown: 'System.TimeoutException' in System.Private.CoreLib.dll
Server timeout (9000.00ms) elapsed without receiving a message from the server.

In the same Closed handler I attempt to reconnect like this:

log("1");
await HubConnection.StopAsync();
log("2");
await Task.Delay(2000);
log("3");
await HubConnection.StartAsync();
log("4");

I only get logged:

1
4

[analogrelay]

You don't need to StopAsync in the closed handler, it's already closed. Not sure why the log messages aren't being displayed. Can you debug and see if any exceptions or such are getting in the way?

On Closed Event after some "A connection with the server could not be established" exceptions I get this:

I think that by dramatically reducing the timeout interval, you've probably made it shorter than the maximum latency of your network and even though the server is trying to ping frequently the messages just aren't quite arriving in time. Is there a reason the default values weren't working for you?

[alexiordan]

Thank you for your time.
Regarding the StopAsync. OK, I'll remove it and see how it works.

Regarding the parameters:
I am just trying to use the same parameters i did with the previous version of SignalR and I was pleased with the results.

One more question. What are the differences between setting the KeepAlive parameter on client-side vs setting it on the server side?

[analogrelay]

The only Keep-Alive that SignalR provides is from Server-to-Client. That is, a SignalR server will ensure that at least one message is sent within the Keep-Alive interval, sending a Ping message if there are no user messages to send.

The "client" Keep-Alive you've configured is part of the options provided to the WebSocket and refers to sending WebSocket Ping frames. Those Ping frames are not visible to the server (they are handled at a lower level) so they have no impact on SignalR's behavior. Those frames only serve to ensure that HTTP proxies sitting in the middle of the connection don't terminate the connection, thinking the other side has timed out. Those frames are sent in both directions (by the server and by the client) automatically, and SignalR has no control over that except to expose the option to configure the duration. Honestly, I think there's very little value (and a cost of increased network traffic) to changing that value from it's default.

[alexiordan]

OK,
Not creating the HubConnection each time, I get eventually into this:

Cannot start a connection that is not in the Disconnected state.

1. I think when I get into this I should re-build the HubConnection. Is this correct?
2. If so, how can I know when should I rebuild the HubConnection and when not to? Can I query its state in some way?

[analogrelay]

Can you confirm what version of SignalR you are using? The preview 1 version should allow you to re-connect without requiring you to rebuild the HubConnection.

[alexiordan]

1. My version is: 1.0.0-preview1-final
2. So that message means that I should rebuild the HubConnection?
3. I'll try to reproduce and write you the steps.

[analogrelay]

So that message means that I should rebuild the HubConnection?

No, it should be working without rebuilding the HubConnection. Versions older than preview 1 did require that you rebuild the connection.

I'll try to reproduce and write you the steps.

That would help a lot. If you're able to narrow down a simple repro we can more easily identify the issue.

[alexiordan]

OK, I've got something.

Thank you for your time and hopefuly I did not make any foolish mistake in using the SignalR in the code below.

I tried not to rebuild the HubConnection and I get after 80 minutes of offline server a breaking exception in client of type System.Runtime.InteropServices.SEHException.

Repro steps:

1. Started client (called StartAsync()) without server online. Which throws me following exception (which I cach to StartAsync()):

Unable to connect to the remote server

2. Started server (after 1 minute or so)
3. Client connected
4. Stopped the server for the entire test sesssion - never restarted (the server only ran for about 1 minute)
5. Client first triggers the Closed event (which I handle to StartAsync() again) with exception parameter:

The remote party closed the WebSocket connection without completing the close handshake.

6. Server is still offline so the StartAsync() throws

Server timeout (9000.00ms) elapsed without receiving a message from the server.

7. From this point on for about 80 minutes I catch this exception and try reconnect.
8. After the 80 minutes the program breaks unrecoverably with following exception:

An unhandled exception of type 'System.Runtime.InteropServices.SEHException' occurred in System.Private.CoreLib.dll
External component has thrown an exception.
The program '[4712] myapp.exe' has exited with code -1073741795 (0xc000001d) 'Illegal Instruction'.

Stack trace is an impressive list with following 4 lines:

--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at myapp.SignalRClient.<StartConnectionAsync>d__35.MoveNext()

repeated over 6700 lines (that is about 1 340 times).

Finally the code that produces this behavior is as follows:

 private void CreateHubConnection() //only called once
        {          
            HubConnection = new HubConnectionBuilder()
                .WithTransport(Microsoft.AspNetCore.Sockets.TransportType.WebSockets)
                .WithWebSocketOptions(a => a.KeepAliveInterval = TimeSpan.FromSeconds(3))
                .WithUrl($"{m_Url}{m_HubName}")
                .WithConsoleLogger()
                .Build();

            HubConnection.ServerTimeout = TimeSpan.FromSeconds(9);
            HubConnection.Closed -= HubConnection_Closed;
            HubConnection.Closed += HubConnection_Closed;
            HubConnection.On("doAction", () => { ActionRequested?.Invoke(this, EventArgs.Empty); });        
        }
  private async Task StartConnectionAsync()
        {
            try
            {
                await HubConnection.StartAsync();              
            }           
            catch (Exception ex)
            {
                await OnConnectionExceptionAsync(ex);
            }
        }
 private async Task OnConnectionExceptionAsync(Exception exception)
        {
            await OnConnectionClosedAsync(exception);
        }
 private async Task OnConnectionClosedAsync(Exception ex)
        {            
            await Task.Delay(2000);
            await StartConnectionAsync();
        }
   private async void HubConnection_Closed(Exception exception)
        {
            await OnConnectionClosedAsync(exception);
        }

[BrennanConroy]

Looks like you're stackoverflowing
StartConnectionAsync -> OnConnectionExceptionAsync -> OnConnectionClosedAsync -> StartConnectionAsync -> OnConnectionExceptionAsync -> etc

[alexiordan]

It is indeed.
I have now changed it to

private async Task StartConnectionAsync()
        {
            try
            {
                if (m_ConnectionIsStarted == false)
                {
                    CreateHubConnection();
                }
             await   HubConection.StartAsync();
                m_ConnectionIsStarted = true;
            }
           
            catch (Exception ex)
            {
                await OnConnectionExceptionAsync(ex);
            }
        }

        private async Task OnConnectionExceptionAsync(Exception exception)
        {
            m_ConnectionIsStarted = false;
            await CleanUpHubConnectionAsync();
            await OnConnectionClosedAsync(exception);
        }

 private async Task CleanUpHubConnectionAsync()
        {
            if (HubConnection != null)
            {
                try
                {
                    await HubConnection.StopAsync();
                }
                catch
                {
                }


                if (HubConnection != null
                {
                    try
                    {
                        HubConnection.Closed -= HubConnection_Closed;
                        await HubConnection.DisposeAsync();
                    }
                    catch
                    {
                    }
                    HubConnection = null;
                }
            }
        }

and let it run.
Do you think should be better now?
I Debug Print on almost each line (removed in the code here) so hopefully will have something later.

[alexiordan]

Sorry! I now realize that updated approach is full of cycles again (theresuch I got into an OutOfMemoryException this time :-) )

So, back to my initial question: What is the proper way to reconnect?

I can try to set a timer and on Elapsed event to evaluate a variable set on ClosedHandler or on exception catch of the StartAsync. Upon evaluation to StartAsync or do nothing until next Elapsed event.

I think is a common scenario for the client to attempt reconnecting to server...

[analogrelay]

You should be able to simply call HubConnection.StartAsync() from inside the HubConnection.Closed event handler.

What is the primary problem you're seeing? From the looks of it, if you're seeing OutOfMemoryException or StackOverflowException errors in your code, it would appear that you are retrying indefinitely and continuing to encounter errors. What's the exception that's occurring during StartAsync to cause the retry loop? It's possible it's an unrecoverable error some kind.

In general, infinite retry is like an infinite loop, if a persistent error develops (network connectivity issues, etc.) you'll retry forever. You need to limit retries and at a certain point you have to give up and report the error to the user.