Automatically retry calls on 429 rate limiting

[dblock]

Followup from #167

[icybin]

Follow up from #167 (comment)

Thanks @dblock . Now I can't deploy my bot on AWS EC2 instance as the network there is so fast. Is there any work-around and/or quick patch?

Thanks a lot.

[icybin]

Logs

I, [2017-09-20T07:13:53.972953 #30810]  INFO -- : post https://slack.com/api/rtm.connect
D, [2017-09-20T07:13:53.973049 #30810] DEBUG -- request: Accept: "application/json; charset=utf-8"
User-Agent: "Slack Ruby Client/0.9.1"
Content-Type: "application/x-www-form-urlencoded"
I, [2017-09-20T07:13:54.764372 #30810]  INFO -- Status: 200
D, [2017-09-20T07:13:54.764476 #30810] DEBUG -- response: content-type: "application/json; charset=utf-8"
content-length: "304"
connection: "close"
access-control-allow-origin: "*"
cache-control: "private, no-cache, no-store, must-revalidate"
date: "Wed, 20 Sep 2017 07:13:54 GMT"
expires: "Mon, 26 Jul 1997 05:00:00 GMT"
pragma: "no-cache"
referrer-policy: "no-referrer"
server: "Apache"
strict-transport-security: "max-age=31536000; includeSubDomains; preload"
vary: "Accept-Encoding"
x-accepted-oauth-scopes: "rtm:stream,client"
x-content-type-options: "nosniff"
x-oauth-scopes: "identify,read,post,client,apps"
x-slack-backend: "h"
x-slack-req-id: "01234567"
x-xss-protection: "0"
x-cache: "Miss from cloudfront"
via: "1.1 1e54e13a0235f358c0b6cad4a71caa42.cloudfront.net (CloudFront)"
x-amz-cf-id: "01234567"
D, [2017-09-20T07:13:54.780978 #30810] DEBUG -- Slack::RealTime::Concurrency::Eventmachine::Socket#connect!: Slack::RealTime::Concurrency::Eventmachine::Client
I, [2017-09-20T07:13:54.781673 #30810]  INFO -- : post https://slack.com/api/rtm.connect
D, [2017-09-20T07:13:54.781727 #30810] DEBUG -- request: Accept: "application/json; charset=utf-8"
User-Agent: "Slack Ruby Client/0.9.1"
Content-Type: "application/x-www-form-urlencoded"
I, [2017-09-20T07:13:55.084847 #30810]  INFO -- Status: 200
D, [2017-09-20T07:13:55.084990 #30810] DEBUG -- response: content-type: "application/json; charset=utf-8"
content-length: "303"
connection: "close"
access-control-allow-origin: "*"
cache-control: "private, no-cache, no-store, must-revalidate"
date: "Wed, 20 Sep 2017 07:13:54 GMT"
expires: "Mon, 26 Jul 1997 05:00:00 GMT"
pragma: "no-cache"
referrer-policy: "no-referrer"
server: "Apache"
strict-transport-security: "max-age=31536000; includeSubDomains; preload"
vary: "Accept-Encoding"
x-accepted-oauth-scopes: "rtm:stream,client"
x-content-type-options: "nosniff"
x-oauth-scopes: "identify,read,post,client,apps"
x-slack-backend: "h"
x-slack-req-id: "01234567"
x-xss-protection: "0"
x-cache: "Miss from cloudfront"
via: "1.1 e92385e75ee2801810ba61a6a2b0eaa8.cloudfront.net (CloudFront)"
x-amz-cf-id: "01234567"
D, [2017-09-20T07:13:55.119388 #30810] DEBUG -- Slack::RealTime::Concurrency::Eventmachine::Socket#connect!: Slack::RealTime::Concurrency::Eventmachine::Client
I, [2017-09-20T07:13:55.120070 #30810]  INFO -- : post https://slack.com/api/rtm.connect
D, [2017-09-20T07:13:55.120124 #30810] DEBUG -- request: Accept: "application/json; charset=utf-8"
User-Agent: "Slack Ruby Client/0.9.1"
Content-Type: "application/x-www-form-urlencoded"
I, [2017-09-20T07:13:55.389582 #30810]  INFO -- Status: 429
D, [2017-09-20T07:13:55.389699 #30810] DEBUG -- response: content-type: "application/json; charset=utf-8"
content-length: "34"
connection: "close"
access-control-allow-origin: "*"
cache-control: "private, no-cache, no-store, must-revalidate"

[dblock]

Your problem looks different @kyanh-huynh, you have 3 rtm.connect calls in there, it's trying to do it again and again, there should be only one. Check your code, you're probably swallowing an exception and retrying continuously.

[icybin]

Thanks @dblock. I put the bot block inside an EventMachine. The idea comes from your comment slack-ruby/slack-ruby-bot#49 (comment)

EventMachine.run do
  MyBot.run
end

[icybin]

It's really fun now

1/ When I first deployed my bot on Ec2 machine, the logs were the same as above. Noted the line Slack::RealTime::Concurrency::Eventmachine::Socket#connect!: that could be something wrong with EventMachine.
2/ Then I disable the EventMachine block, simply use MyBot.run, then my bot can connect to Slack API server
3/ The I enable the EventMachine block again, EventMachine.run {MyBot.run}, and now the bot can also connect. Now I really see Slack::RealTime::Concurrency::Celluloid::Socket#connect!: WebSocket::Driver::Client, no more Slack::RealTime::Concurrency::Eventmachine::Socket#connect!:

Any ideas?

** Update: ** It seems the problem happens randomly. There must be something wrong with Ruby gems on my Bot server. Let me check :(

[dblock]

I would 1) switch to Celluloid 2) put a begin/rescue around .run, are you getting an exception that you're not seeing?

[icybin]

Hi @dblock ,

Thanks for your support. After some random tests, I can come to a final solution:
1/ On my laptop (archlinux, ruby 2.4.1), I have to put MyBot.run between an EventMachine block.
2/ On Bot server (ubuntu 16.04, ruby 2.3.1), I have to remove the EventMachine block and simply use MyBot.run.

I have some long running task, and on my laptop it seems that 1/ is the only working option.

If you think the case is interesting, please let me know I can give more debug messages. I think there is some difference between two servers, but I'm not sure. (I used Bundle/Gemfile.lock)