Skip to content
This repository has been archived by the owner on Jun 30, 2022. It is now read-only.

Adapter disconnects after a period of time #67

Closed
Ginja opened this issue Dec 11, 2015 · 20 comments
Closed

Adapter disconnects after a period of time #67

Ginja opened this issue Dec 11, 2015 · 20 comments

Comments

@Ginja
Copy link
Contributor

Ginja commented Dec 11, 2015

lita-slack 1.7.2
lita 4.6.1

In the last couple of days our lita bot has been disconnecting from Slack every couple of minutes. I've also noticed that when it comes back up, it will only respond to one or two commands before it stops responding to anything (yet it's still connected). After about 3-5 minutes it will then just simply disconnect from a close event:

[2015-12-11 16:58:13 UTC] INFO: Closing because of #<Faye::WebSocket::API::CloseEvent:0x00000000be7328>
[2015-12-11 16:58:13 UTC] INFO: Disconnected from Slack.

I've used pry-remote to take a look at that Close Event and it's seems normal. In fact, all that was in the buffer was:

@buffer="\x81A{\"type\":\"presence_change\",\"user\":\"USERID\",\"presence\":\"active\"}\x81\x02{}", @offset=71>,

I've tried disabling all of our plugins, but the same behaviour occurs. My guess is that the lita-slack adapter is being rate-limited, and subsequently terminated, but I couldn't find a response to back that hypothesis up.

I also tried to turn debugging on for the Faye::WebSocket::Client class, but couldn't figure out where to specify it aside from the options hash. And since { debug: true } is not a valid option, lita crashed instantly.

Any help in debugging this issue would much be appreciated! Thanks :)
@jimmycuadra
Copy link
Contributor

What a frustrating issue. This is the first I'm hearing of it. I wonder if other people are experiencing this and haven't reported it, or if there's something unique to your setup. In either case, seems like the adapter should provide more debugging information, perhaps leveraging any ability Faye has to let you log the websocket activity.

@Ginja
Copy link
Contributor Author

Ginja commented Dec 11, 2015

@jimmycuadra that would be great. I've also opened up an issue with Slack and they're looking at their logs. I'll update this PR if they come up with anything.

@pseidemann
Copy link

I experience the same issues.

here is the log with log level debug:

[2015-12-14 18:10:48 UTC] DEBUG: Connecting to the Slack Real Time Messaging API.
[2015-12-14 18:10:48 UTC] DEBUG: Connected to the Slack Real Time Messaging API.
[2015-12-14 18:10:49 UTC] INFO: Connected to Slack.
[2015-12-14 18:10:49 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:10:49 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:11:04 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:11:44 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:11:48 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:11:51 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:11:56 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:12:00 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:12:01 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:12:01 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:12:03 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:12:03 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 18:12:04 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:12:05 UTC] DEBUG: Dispatching message to Lita from UXXXX.
[2015-12-14 18:12:12 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:12:13 UTC] DEBUG: Dispatching message to Lita from UYYYY.
[2015-12-14 18:12:32 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 18:13:45 UTC] DEBUG:  event received from Slack and will be ignored.
[2015-12-14 18:13:45 UTC] INFO: Disconnected from Slack.
[2015-12-14 18:13:45 UTC] DEBUG: Closing connection to the Slack Real Time Messaging API.

I never see messages sent by response.reply(...) from the bot and it disconnects after a couple of minutes. it worked just fine until 2015-12-14 12:30:00 UTC.

in the log there is one interesting line:

[2015-12-14 18:13:45 UTC] DEBUG:  event received from Slack and will be ignored.

it seems like it gets some event it can't parse. maybe this is related?

@NikolayRys
Copy link
Contributor

me too:

$ bundle exec lita
[2015-12-15 05:05:43 UTC] INFO: Connected to Slack.
[2015-12-15 05:15:52 UTC] INFO: Disconnected from Slack.

@m-funky
Copy link

m-funky commented Dec 15, 2015

me too

[2015-12-14 05:53:20 UTC] DEBUG: Connecting to the Slack Real Time Messaging API.
[2015-12-14 05:53:21 UTC] DEBUG: Connected to the Slack Real Time Messaging API.
[2015-12-14 05:53:21 UTC] INFO: Connected to Slack.
[2015-12-14 05:53:21 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:53:21 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:53:21 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:53:53 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:53:54 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:54:12 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:55:03 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:55:08 UTC] DEBUG: user_typing event received from Slack and will be ignored.
[2015-12-14 05:55:10 UTC] DEBUG: Dispatching message to Lita from UXXXXXX.
[2015-12-14 05:55:50 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:55:52 UTC] DEBUG: presence_change event received from Slack and will be ignored.
[2015-12-14 05:56:03 UTC] INFO: Disconnected from Slack.
[2015-12-14 05:56:03 UTC] DEBUG:  event received from Slack and will be ignored.
[2015-12-14 05:56:03 UTC] DEBUG: Closing connection to the Slack Real Time Messaging API.

from 12/10

@jimmycuadra
Copy link
Contributor

For everyone replying saying they are experiencing this issue, which of these is true for you?

  1. You've been using Lita for a while and this started happening recently.
  2. You've been using Lita for a while and this has always happened.
  3. You just started using Lita and this has always happened.

Where "a while" is somewhere on the order of many weeks or more.

@NikolayRys
Copy link
Contributor

It's the third one in my case.

@m-funky
Copy link

m-funky commented Dec 15, 2015

My case is 1.
I've been using Lita for 10 months.

@Ginja
Copy link
Contributor Author

Ginja commented Dec 15, 2015

1 as well.

@pseidemann
Copy link

1 for me

@pseidemann
Copy link

my bot still can't send any message since monday. I've implemented my own handlers like in this example http://docs.lita.io/plugin-authoring/handlers/#examples, response.reply() gets called with a proper string but nothing appears in the slack channel. I didn't changed anything in the code. using the latest lita and lita-slack gems didn't helped.

was there some major change in the slack api or something?

@Ginja
Copy link
Contributor Author

Ginja commented Dec 15, 2015

I've passed along this GitHub issue to Slack's support, and they're passing it along to their engineers. I'll update this thread when they get back to me.

@jimmycuadra
Copy link
Contributor

Thank you, Ginja. As dismissive as it sounds to say "this is probably a Slack issue," it very well may be, so let's see what they say.

@dwradcliffe
Copy link

Me too, situation 1. Lita is receiving messages from slack for a few minutes but no responses ever make it back to Slack. After 2-3 minutes it crashes.

@v-fedorov
Copy link

Having the exact same issue

@maclover7 maclover7 mentioned this issue Dec 19, 2015
Closed
@dwradcliffe
Copy link

Ours started working again. I wonder if something changed on the Slack side?

@pseidemann
Copy link

I can confirm mine started working again, too

@Ginja
Copy link
Contributor Author

Ginja commented Dec 22, 2015

Same here. I updated our ticket to Slack's support and this was their response:

Strange, no changes were made to specifically address this issue but it's possible that something went out that had an effect. I would ask though that if you see this come up again to let me know right away — it always feels a little uneasy when something fixes itself!

@jimmycuadra could we leave this issue open as a reminder to add some debugging options for faye-websocket?

@jimmycuadra
Copy link
Contributor

Yes, definitely. Thanks for following up, everyone!

@jimmycuadra
Copy link
Contributor

I'm gonna close this, but those interested in tracking additional debugging output for the adapter can follow #32.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dwradcliffe @jimmycuadra @pseidemann @Ginja @v-fedorov @NikolayRys @m-funky