Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconnect when websocket closes abnormally #274

Merged
merged 3 commits into from
Jun 16, 2016

Conversation

selfcontained
Copy link
Contributor

I've been debugging some websocket connection issues that we've seen pretty consistently on Beep Boop where the connection lasts 6 or 7 days, then disconnects, and doesn't attempt to reconnect. I've narrowed this down to the following scenario:

  • websocket connection goes stale, i.e. PONG receipt times out
  • attempts to reconnect - but is unsuccessful
  • a close event triggers on the websocket with an error code 1006 with this unsuccessful reconnect attept, meaning it was an abormal close (more info here)

Currently that's where the websocket reconnect attempt stops, so this PR adds another reconnect attempt when an abnormal close event is received on the websocket. We've seen this fix our slack rtm connections going down on a few bots on Beep Boop.

Below are some annotated logs that helped me figure out what was going on:

// Stale RTM Connection via PONG timeout - called closeRTM - ln 165

2016-06-11T05:19:00.160Z - error: RTM Connection closed
  // from `beepboop-botkit` logged on `rtm_close` event

2016-06-11T05:19:01.166Z - botkit: ** BOT ID: roflbot ...reconnect attempt #1 of Infinity being made after 1000ms
  // reconnect() called - ln 77

2016-06-11T05:19:01.181Z - botkit: ** API CALL: https://slack.com/api/rtm.start
  // startRTM() via reconnect()

2016-06-11T05:19:01.527Z - botkit: ** BOT ID: roflbot ...attempting to connect to RTM!
 // startRTM() via reconnect()

2016-06-11T10:25:31.915Z - botkit: RTM close event: 1006 :
  // unsuccessful websocket connection: websocket `close` event

2016-06-11T10:25:31.916Z - error: RTM Connection closed
  // from `beepboop-botkit` logged on `rtm_close` event

2016-06-11T10:25:32.917Z - botkit: ** BOT ID: roflbot ...reconnect attempt #1 of Infinity being made after 1000ms
  // reconnect() attempt

2016-06-11T10:25:32.918Z - botkit: ** API CALL: https://slack.com/api/rtm.start
  // startRTM() via reconnect()

2016-06-11T10:25:33.260Z - botkit: ** BOT ID: roflbot ...attempting to connect to RTM!
 // startRTM() via reconnect()

@benbrown benbrown merged commit 714788d into howdyai:master Jun 16, 2016
@benbrown
Copy link
Contributor

Thanks @selfcontained!! Merged!

@selfcontained selfcontained deleted the close-error-reconnect branch June 16, 2016 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants