New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best way to reconnect? #81

Open
JSteunou opened this Issue Aug 20, 2014 · 25 comments

Comments

Projects
None yet
@JSteunou

JSteunou commented Aug 20, 2014

Hi!

I was wondering what should be the best way to reconnect after losing connection. I'm facing a case where an HTML5 app lose the connection when the mobile is going on hold.

I though calling a connect again in a connection error callback could work, but it does not. Freeze on "Opening Web Socket..."

I managed it by calling for a new connection in connection error callback. But I'm losing all my subscription so I have to store those in order to subscribe again. Not great.

I'm thinking about another thing: is heartbeat could prevent losing connection?

@rfox90

This comment has been minimized.

rfox90 commented Aug 20, 2014

heartbeats are designed to keep a connection open yes.

Just re-connecting should be fine. Any idea why its hanging? What does your broker say?

@JSteunou

This comment has been minimized.

JSteunou commented Aug 21, 2014

No reason, no clue, no log :( I'm using stompjs + sockjs + RabbitMQ.

After testing, heartbeats does not work for my case. On iOS, on hold, the system shut down network connection and others things. I still lose the connection to my broker.

@jmesnil

This comment has been minimized.

Owner

jmesnil commented Aug 22, 2014

what do you mean by "On iOS, on hold"?

On iOS, the network connection will be closed when the application goes into background.

@rfox90

This comment has been minimized.

rfox90 commented Aug 22, 2014

So is the client object being garbage collected?

@JSteunou

This comment has been minimized.

JSteunou commented Aug 22, 2014

@jmesnil iPhone 4s, iOS 6 & 7, when the mobile is on sleep / hibernate (maybe a better word than on hold).

The user use safari or chrome, the application is an HTML application. After a time of inactivity or when pressing the on/off button the mobile goes to sleep / hibernate.

When the user wake up the mobile, the browser is still opened on the application, and the application works well, but the stomp client is disconnected. If I call connect in the error callback it's stuck on "Opening Web Socket..."

@rfox90 I dont think objects are garbage collected. Mobile put on hold everything, even running JavaScript in browser when going to sleep / hibernate but usually all things restart smoothly on wake up.

@sckoh

This comment has been minimized.

sckoh commented Sep 10, 2015

have you solved this?

@JSteunou

This comment has been minimized.

JSteunou commented Sep 10, 2015

Yes, I manually store all topics I subscribed to and when I catch a socket error I re-create a connection and re-subscribe to all my topics.

@bfwg

This comment has been minimized.

bfwg commented Sep 13, 2016

My implementation, try to reconnect every 1 second after disconnected.

let ws = new WebSocket(socketUrl);
let client = webstomp.over(ws);
client.connect({}, (frame) => {
  successCallback();
}, () => {
  reconnect(socketUrl, successCallback);
});

function reconnect(socketUrl, successCallback) {
  let connected = false;
  let reconInv = setInterval(() => {
    ws = new WebSocket(socketUrl);
    client = webstomp.over(ws);
    client.connect({}, (frame) => {
      clearInterval(reconInv);
      connected = true;
      successCallback();
    }, () => {
      if (connected) {
        reconnect(socketUrl, successCallback);
      }
    });
  }, 1000);
}

@picarro-sgupta

This comment has been minimized.

picarro-sgupta commented Dec 7, 2016

How you are making sure that you are not losing any messages when you "re-create a connection and re-subscribe to all my topics."

@JSteunou

This comment has been minimized.

JSteunou commented Dec 7, 2016

@basz

This comment has been minimized.

basz commented Feb 8, 2017

@mouss-delta

This comment has been minimized.

mouss-delta commented Jun 27, 2017

@bfwg thanx for your solution, it works like a charm.

However I have a question, in reconnect function, in error callback I don't understand why this is if (connected) and not if (!connected), I have tried with if (!connected) but the socket keeps reconnecting even if it is already connected.

@bfwg

This comment has been minimized.

bfwg commented Jun 27, 2017

Hey @mouss-delta , the connected is the flag to prevent a recursive hell. The logic is: we only want to call the reconnect function when there is a successful connection.

@mouss-delta

This comment has been minimized.

mouss-delta commented Jun 27, 2017

@bfwg thanx a lot for your quick answer I understand now :)

@bfwg

This comment has been minimized.

bfwg commented Jun 27, 2017

@mouss-delta you are welcome. The code I'm using in my code base currently looks like the below, as you can see the logic is much cleaner.

  let ws = new WebSocket(socketUrl);
  let client = webstomp.over(ws);

  connectAndReconnect(socketInfo, successCallback);

  private connectAndReconnect(socketInfo, successCallback) {
    ws = new WebSocket(socketUrl);
    client = webstomp.over(ws);
    client.connect({}, (frame) => {
      successCallback();
    }, () => {
      setTimeout(() => {
        this.connectAndReconnect(socketInfo, successCallback);
      }, 5000);
    });
  }
@brendanbenson

This comment has been minimized.

brendanbenson commented Jun 27, 2017

@bfwg Can you explain the use for the socketInfo parameter to connectAndReconnect? I see it being passed in, but it's never used.

@bfwg

This comment has been minimized.

bfwg commented Jun 27, 2017

@brendanbenson it's just the ws URL.

@mouss-delta

This comment has been minimized.

mouss-delta commented Jun 29, 2017

Hi, @bfwg avec you had any issue with socket reconnexion on Chrome. The given algorithm works just fine with firefox but non in Chrome, it is like the error callback of client.connect is not even taken into account

@bfwg

This comment has been minimized.

bfwg commented Jun 29, 2017

Hey, @mouss-delta no the algorithm works fine for me on Chrome.

@kodmanyagha

This comment has been minimized.

kodmanyagha commented Nov 1, 2017

File: node_modules/@stomp/stompjs/lib/stomp.js
Line: 352
There is a method name is onclose for websocket. In line 352 it's handling this. Real method is:

this.ws.onclose = (function(_this) {
  return function() {
    var msg;
    msg = "Whoops! Lost connection to " + _this.ws.url;
    if (typeof _this.debug === "function") {
      _this.debug(msg);
    }
    _this._cleanUp();
    if (typeof errorCallback === "function") {
      errorCallback(msg);
    }
    return _this._schedule_reconnect();
  };
})(this);

We can access this from our app. I'm writing here my usage:

connect() {
	var self = this;

	//self.socket = new SockJS(self.getConnectionUrl());
	var token = window.myApp.getCookie("token");

	self.socket = new SockJS(self.getChatWebsocketConnectionUrl());

	self.socket.onheartbeat = function() {
		console.log("heartbeat");
	};

	self.stompClient = Stomp.over(self.socket);
	self.stompClient.connect(self.getDefaultHeaders(), function(frame) {
		self.setConnected(true);
		console.log("Connected: " + frame);
	});

	// here is important thing. We must get current onclose function for call it later.
	var stompClientOnclose = self.stompClient.ws.onclose;
	self.stompClient.ws.onclose = function() {
		console.log("Websocket connection closed and handled from our app.");
		self.setConnected(false);
		stompClientOnclose();
	};
}

If websocket server down or a network problem occured than we can handle close event.

@super86

This comment has been minimized.

super86 commented Jan 15, 2018

let ws = new WebSocket(socketUrl);
let client = webstomp.over(ws);

connectAndReconnect(socketInfo, successCallback);

private connectAndReconnect(socketInfo, successCallback) {
ws = new WebSocket(socketUrl);
client = webstomp.over(ws);
client.connect({}, (frame) => {
successCallback();
}, () => {
setTimeout(() => {
this.connectAndReconnect(socketInfo, successCallback);
}, 5000);
});
}

when my code is like this. I shutdown the server process,the errorcallback is called twice,what's the problem??@bfwg

@GoranGjorgievski

This comment has been minimized.

GoranGjorgievski commented Feb 1, 2018

@super86 The problem with this function is that, it is possible the Websocket's CONNECTING phase might last more than 1 second. In this case another setInterval will be called/initialized, and if the process repeats it might come to multiple setIntervals instead of one.

What I did was the adding two checks inside setInterval ( in your case inside connectAndReconnect), so it looks like like this:

var reconInt = setInterval(function() {
           if (client.ws.readyState === client.ws.CONNECTING) {
               return;
           }

           if (client.ws.readyState === client.ws.OPEN) {
               clearInterval(reconInt);
               return;
           }

           var ws = new WebSocket(config.stomp.url);

           client = new Stomp.over(ws);

           client.heartbeat.outgoing = 30000;
           client.heartbeat.incoming = 30000;
           client.debug = null;

           client.connect(
               credentials,
               () => {
                   clearInterval(reconInt);
                   connected = true;
                   parent.connectCallback(callbackParams, client, accountUid);
               },
               () => {
                   if (connected) {
                       parent.errorCallback(
                           callbackParams,
                           config,
                           client,
                           credentials,
                           accountUid,
                           parent
                       );
                   }
               }
           );
       }, 1000);
   }

Notice the following part at the top:

if (client.ws.readyState === client.ws.CONNECTING) {
                return;
            }

if (client.ws.readyState === client.ws.OPEN) {
        clearInterval(reconInt);
        return;
}

Here in the first part I am checking whether the previous setInterval client status in in CONNECTING phase state, which is possible if the client.connect lasts more than 1 second. In this case it is still possible that the CONNECTING phase fails, hence I am not removing the interval but having a second check.
Here I am checking if the state is open, meaning some previous interval already opened a connection, then clear the interval and exit from the current one.

You can also check the code here: https://github.com/inplayer-org/inplayer.js/blob/master/src/Socket/index.js

@GoranGjorgievski

This comment has been minimized.

GoranGjorgievski commented Feb 5, 2018

How do you guys/girls handle the fact that if you have a lot of concurrent users and Stomp fails at some point, every single one of them will try to reconnect at the same time and the server will probably spam and block the servers?

We had just 6k concurrent users, and are using Stomp over WebSocket + RabbitMQ. However during an event, there might've been some interruption and the WS connection failed for all of them. Afterwards all the users flooded our servers with the 1 second callback to reconnect(we had like a lot of requests pending and stacked), so we had to restart everything.

One solution I can think of is to have something like a Fibonacci sequence for the setInterval(), time call, e.g. -> setTimeout(). But still, the initial 1-5 seconds a lot of requests will come at once. Plus the fact that I would have to have an infinite loop blocking everything, util it connects...

Anyone has any better idea to optimize this?

@milosonator

This comment has been minimized.

milosonator commented Feb 6, 2018

@GoranGjorgievski Basically you need to add a random amount of starting delay + a backoff delay.
For instance, you randomize the initial wait time somewhere between 5-30 seconds, after which you will do exponential backoff (or fib, if you please). Make sure to test your server behavior in a heavy load case to make sure it will eventually recover.

Even better (and related to the solution I'm looking for) the server should be able to close connections with a certain close code, and then the client can reconnect accordingly (server can tell the client how long to wait, for instance).

@GoranGjorgievski

This comment has been minimized.

GoranGjorgievski commented Feb 6, 2018

@milosonator Yes thanks! Thanks what I actually did, having a random starting setTimeout + something like incremental 'Fibonacci like' timed callback: https://github.com/inplayer-org/inplayer.js/blob/master/src/Socket/index.js

errorCallback(
        ...,
        timeoutStart = 0
    ) {
        ...
        if (timeoutStart === 0) {
            timeoutStart =
                (Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) * 1000; //get a random start timeout between 1-max
        }
        setTimeout(function() {
            if (
                client.ws.readyState === client.ws.CONNECTING ||
                client.ws.readyState === client.ws.OPEN
            ) {
                return;
            }
            var ws = new WebSocket(config.stomp.url);

            client = new Stomp.over(ws);
            ...
            client.connect(
                credentials,
                () => {
                    parent.connectCallback(callbackParams, client, accountUid);
                    //reset the timeoutStart
                    timeoutStart =
                        (Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) *
                        1000; //get a random start timeout between 1-max
                },
                () => {
                    parent.errorCallback(
                        ...
                        timeoutStart
                    );
                }
            );
        }, timeoutStart);
        if (timeoutStart >= MAX_INTERVAL) {
            //if more than 10 minutes reset the timer
            timeoutStart =
                (Math.floor(Math.random() * MAX_INITIAL_INTERVAL) + 1) * 1000; //get a random start timeout between 1-max
        } else {
            timeoutStart += Math.ceil(timeoutStart / 2);
        }
    }

In addition, I also added a TOP limit for the timeout for 10 minutes.
e.g. If the timeout interval reaches 600000 milliseconds, it should reset back again to the random initial number. This way I am avoiding having some unreasonable high re-connection times for the client.

Still in testing phases and thinking about somehow getting the number of active socket connections on OPEN, and then when it fails just scale the random MAX_INITIAL_INTERVAL number depending on that number of connections. Once I manage to improve it, I will make sure to post it here of course.

@scionaltera scionaltera referenced this issue Jul 25, 2018

Closed

Idle timeout #65

yurloc added a commit to kiegroup/optaweb-vehicle-routing that referenced this issue Oct 8, 2018

PLANNER-1275 Automatically reconnect after backend restart
Useful resources:

- jmesnil/stomp-websocket#81 (comment)
  Simple reconnection solution based on setTimeout(), some other
  interesting ideas later in the discussion.

- https://github.com/JSteunou/webstomp-client
  webstomp-client readme

- http://jmesnil.net/stomp-websocket/doc/
  The original STOMP client documentation (webstomp-client is a fork of
  stomp-websocket)

yurloc added a commit to kiegroup/optaweb-vehicle-routing that referenced this issue Oct 8, 2018

PLANNER-1275 Automatically reconnect after backend restart
Useful resources:

- jmesnil/stomp-websocket#81 (comment)
  Simple reconnection solution based on setTimeout(), some other
  interesting ideas later in the discussion.

- https://github.com/JSteunou/webstomp-client
  webstomp-client readme

- http://jmesnil.net/stomp-websocket/doc/
  The original STOMP client documentation (webstomp-client is a fork of
  stomp-websocket)

Fixes #3.

yurloc added a commit to kiegroup/optaweb-vehicle-routing that referenced this issue Oct 12, 2018

PLANNER-1275 Automatically reconnect after backend restart
Useful resources:

- jmesnil/stomp-websocket#81 (comment)
  Simple reconnection solution based on setTimeout(), some other
  interesting ideas later in the discussion.

- https://github.com/JSteunou/webstomp-client
  webstomp-client readme

- http://jmesnil.net/stomp-websocket/doc/
  The original STOMP client documentation (webstomp-client is a fork of
  stomp-websocket)

Fixes #3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment