Hitting 429 on a large channel with files

[rusq]

Telegram channel:

hi there, thought I’d ask - in v1.1.1/latest - it looks like in large channels, when I get ratelimited by slack, slackdump seems to return the following:

2022/01/18 23:59:13 channel “xxxx”: slack rate limit exceeded, retry after 1s
2022/01/18 23:59:13 job finished, dumped 0 channels

We should either die with error or retry, and we do neither.

Follow up:

Initially tried to dump multiple channels but switched to one by one, still ran into the ratelimuting after waiting roughly an hour. It seems to vary how quickly I get rate limited, sometimes 1000 messages, sometimes 500

[foucist]

I'm also getting this error when I'm getting to 1800 messages on a channel I'm trying to dump. It would be nice if it could write whatever it had to a file instead of just exiting.

By the way I tried changing the tier values & recompiling, but even at 1/10/25/50 (tier1/tier2/tier3/tier4 values) it still throws the same error "slack rate limit exceeded, retry after 1s" after 1800 messages.

It's interesting that a 429 seems to get triggered after the same amount of messages, regardless of the timing.

I guess the "1s" message is coming from the slack-go library? The https://api.slack.com/docs/rate-limits#rate-limits__responding-to-rate-limiting-conditions implies it would normally be asking for a retry after 30 seconds.

[rusq]

Hey @foucist thanks for confirming, yes, this message is from the slack library.

It's interesting that it hits the rate limit after 1800 messages square.

I suspect that it is because of there was no limiter on files downloads, so now file downloader and message dumper use the same limiter instance After refreshing the memory in the rate limit doc, and discovering that the rate limiting is per method, it seems that files are not an issue here, as the library just sends the Get request, instead of polling any of the API endpoints for that. I think what happens here is that by the time it reaches 1800 messages, our internal limiter drifts away from slack rate limiter, (that might be because the same limiter was incorrectly used for 2 different api methods) and therefore we get this "retry after 1s" error. I wrapped the calls with the retry function that will retry the API call should it see the RateLimitedError. Also, now each method has it's own limiter. (see #13).

Would you be able to check, if you compile the branch "rate-limit" would it be able to complete the download of that channel that currently fails for you? That would be great help in confirming this.