Cannot connect to Slack: "Error (code 0): ratelimited"

[bmasonrh]

Today I cannot log into Slack through weechat/wee-slack. The following error is presented:

Failed requesting rtm.connect with token
xoxc-[redacted], retrying. If this persists, try
increasing slack_timeout. Error (code 0): ratelimited

Two attempts are made at 60-second intervals, after which the message becomes:

Failed requesting rtm.connect with token
xoxc-[redacted], will not retry after too many failed 
attempts. If this persists, try increasing slack_timeout. 
Error (code 0): ratelimited

and nothing else happens.

Things that I've tried but didn't help:

• Getting a new token
• Updating to the latest update with commit c303a77 ("Include cookie when making the websocket connection")
• Changing slack_timeout to 90000
• Switching my outbound IP address (my WiFi and wired connections use different ISPs)

I added the following change so I could see the value of retry_after:

$ git diff
diff --git a/wee_slack.py b/wee_slack.py
index 84c437f..b0e5e5d 100644
--- a/wee_slack.py
+++ b/wee_slack.py
@@ -680,7 +680,7 @@ class EventRouter(object):
                 if name.lower() == "retry-after":
                     retry_after = int(value.strip())
                     request_metadata.retry_time = time.time() + retry_after
-                    return "", "ratelimited"
+                    return "", "ratelimited, retry after "+str(retry_after)+"s"
 
         return body, ""

The new message indicated that retry_after is 60 seconds.

[bmasonrh]

Hm. Seems to be working now. If it continues to work for the rest of the day, I'll close this.

[trygveaa]

Are you sure you had upgraded and waited for the ratelimit to pass? This is the issue that should be fixed with the latest commit (c303a77).

[bmasonrh]

Yes, I verified that I had upgraded. I checked the code in ~/.local/share/weechat/python/slack.py to make sure that c303a77 had been applied. How long should it have taken the ratelimit to pass? The value of retry_after was 60 seconds.

[trygveaa]

Hm, weird. It takes the value of retry_after for the ratelimit to pass, so it should have worked after 60 seconds. Did you continuously get the Failed requesting rtm.connect message every 60 seconds after upgrading too?

What happened before that commit is that connecting to the websocket failed, so it immediately tried connecting again by requesting rtm.connect (which gives the link to the websocket endpoint) and connecting to the websocket again. Since all connections to the websocket failed this happened indefinitely, but after a few loops it gets ratelimited and then it waits for the rate limit to try again.

So if this still happened for you what to check would be to call the rtm.connect endpoint and trying to connect to the websocket and see if that fails. If it works now I don't think there's anything to do unless it starts failing again though.

Here is a command to call the rtm.connect endpoint:

curl -H 'Cookie: d=<d-cookie>' -H 'Authorization: Bearer <token>' https://api.slack.com/api/rtm.connect

[bmasonrh]

The issue seems to be back. I just received the following message:

Lost connection to slack team <team-name>.slack.com (no pong), reconnecting.

followed by

Failed requesting rtm.connect for team 
domain=<team-name> nick=<my-nick>, retrying. If 
this persists, try increasing slack_timeout. 
Error (code 0): ratelimited, will retry in 120s

The 120-second retry iis because I tweaked my fork of the code a bit to see if increasing the retry time would help (it didn't):

$ git diff
diff --git a/wee_slack.py b/wee_slack.py
index 84c437f..3ffb38e 100644
--- a/wee_slack.py
+++ b/wee_slack.py
@@ -679,8 +679,8 @@ class EventRouter(object):
                 name, value = header.split(":", 1)
                 if name.lower() == "retry-after":
                     retry_after = int(value.strip())
-                    request_metadata.retry_time = time.time() + retry_after
-                    return "", "ratelimited"
+                    request_metadata.retry_time = time.time() + 2*retry_after
+                    return "", "ratelimited, will retry in " + str(2*retry_after) + "s"
 
         return body, ""

[bmasonrh]

Hm, weird. It takes the value of retry_after for the ratelimit to pass, so it should have worked after 60 seconds. Did you continuously get the Failed requesting rtm.connect message every 60 seconds after upgrading too?

Yes.

Working on trying the curl command. I'm assuming <token> is the value from python.slack.slack_api_token in .config/weechat/plugins.conf. Where does the value of <d-cookie> come from?

[trygveaa]

The format of python.slack.slack_api_token is <token>:<d-cookie> (when using session/xoxc tokens) so take each part separately from it.

[bmasonrh]

I manually grabbed the token:cookie using the instructions in README.md and it seems to be working again. I suspect that there may be an issue with the script that I use to grab the token from the browser (we have a home-grown script that automatically launches the browser and extracts the token/cookie).

[trygveaa]

Great, let's hope it continues working now then.

Have you tried the extract_token_from_browser.py script in this repo for extracting the token from your browser?

[tonivade]

I'm having a similar issue.

This the content on the log file:

2023-09-19 16:05:36             ERROR: Failed connecting to Slack with token xoxs-?????????: channels: invalid_auth, channels: invalid_auth, members: invalid_auth, usergroups: invalid_auth, prefs: invalid_auth, presence: invalid_auth
2023-09-19 16:06:33             Failed requesting rtm.connect for team domain=??? nick=???, retrying. If this persists, try increasing slack_timeout. Error (code 0): ratelimited
2023-09-19 16:07:36             Failed requesting rtm.connect for team domain=??? nick=???, retrying. If this persists, try increasing slack_timeout. Error (code 0): ratelimited
2023-09-19 16:08:39             Failed requesting rtm.connect for team domain=??? nick=??, retrying. If this persists, try increasing slack_timeout. Error (code 0): ratelimited

I tried to regenerate de token following the instructions, but it seems the problem persists.

I tried to set python.slack.background_load_all_history to "false" and increase the value slack_timeout but without success.

[trygveaa]

@tonivade: Are those four lines from the same load of wee-slack? I can't really see how you could get invalid_auth first, and then ratelimited from rtm.connect as it shouldn't continue to call rtm.connect if you get invalid_auth.

Tokens starting with xoxs- has been gradually phased out by Slack and has stopped working at various times for different people. So now you need to get a token that starts with xoxc- and this token also requires that you include the cookie named d and in some cases the cookie named d-s. They should be specified in the format <token>:d=<d_cookie>;d-s=<d-s_cookie> as described in https://github.com/wee-slack/wee-slack#get-a-session-token. Have you made sure to include the cookies with the new token?

[tonivade]

I generated again the token, the error Failed connecting to Slack with token xoxs-????????? doesn't appear anymore. But now I can see this in the logs:

11:48:05 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:05 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:14 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:14 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:15 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:16 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:18 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:18 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:21 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:21 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:24 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:24 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:27 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:27 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:30 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:30 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:33 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:33 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:36 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:48:36 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.
11:48:39         weechat | Failed requesting rtm.connect for team domain=xxxxxxx nick=xxxxxxxx, retrying. If this persist
                         | try increasing slack_timeout. Error (code 0): ratelimited
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
11:49:25 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on send), reconnecting.
11:49:40 slack.xxxxxxx | Connected to Slack team XXXXXXXXX (xxxxxxx.slack.com) with username xxxxxxxx
11:49:40 slack.xxxxxxx | Lost connection to slack team xxxxxxx.slack.com (on receive), reconnecting.

and it repeats all the time: connection lost then failed requesting rtm.connect.

[tonivade]

It seems using the latest version from master fix the issue. This commit: c303a77

Thank you for your support.

[r-vdp]

Could we consider making a release including this fix so that we can more easily get this into distribution repositories?

[trygveaa]

@r-vdp: Yes! I'll make a new release a bit later today.

[trygveaa]

Version 2.10.1 has now been released with the fix.

It looks like the issue here is solved with this, so I'll close the issue.