Twitter API throttling requests (429 Too Many Requests)

[ToniPortal]

Hello, when I use certain scraper functions it gives me this error

      status: 429,
      statusText: 'Too Many Requests',

It was mostly to ask do you know when it's going to stop saying that ?
Or will it be fixed ?

[karashiiro]

I haven't seen that error from this before, but I'm currently working on porting over a month's worth of commits (#9, #6), so once that's done this might be fixed.

[karashiiro]

I just released v0.3.0 which ports over the most critical changes, let me know if that helps.

[karashiiro]

Actually, I just noticed the repo tests are failing with 429, looks like a regular rate limit. I'll need to set up a backoff for this.

[karashiiro]

Should be fixed with v0.3.1 (#12).

[karashiiro]

Made a minor adjustment in v0.3.2 btw, v0.3.1's backoff can get stuck in a retry loop for a while sometimes, which the latest version fixes.

[karashiiro]

Apparently that wasn't what was happening - the API actually just rate-limits the client for 14 minutes after a certain point. I updated the throttling mechanism to handle this, but I'm not sure if there's any real way to handle this beyond the delay.

[ToniPortal]

Oh thanks for the clarification!
Really not cool to have put so many limitations on twitter, hope one day they change all these limitations they put in place...

[ImTheDeveloper]

Is the restriction on IP address or just the tokens being used? If its the token can you regenerate the auth and carry on? Given we approximately know the limits of each endpoint (50 requests per 15 minutes on tweets/replies) we could count the usage of a token and generate before. I see in the code already it takes into account the token validity over x time so this would be now based on usage too.

Edit: Specifically in regards to guest token usage. I expect using a fully authed account would limit the consumer.

[karashiiro]

I just gave that a try, and it doesn't work - as soon as you try to get a new guest token you get rate-limited.

diff for reference: diff.txt