sleep-on-rate-limit not working

[ctlewitt]

I initialized with sleep_on_rate_limit set to true, like this:
api = twitter.Api(consumer_key = CONSUMER_KEY,
consumer_secret=CONSUMER_SECRET,
access_token_key=ACCESS_TOKEN,
access_token_secret=ACCESS_SECRET,
sleep_on_rate_limit=True)
I have a long list of users I am iterating through, and I'm getting their tweets from the last week using the GetSearch command:
results = api.GetSearch(term="from:"+user,
since_id=sincetweet,
max_id=maxtweet,
count=10,
result_type="recent")
I was expecting my program to sleep while waiting for Twitter's rate limit to refresh, but it instead pauses for a few minutes and then throws a "Rate limit exceeded" code: 88 error, like this:
Traceback (most recent call last):
File ".../collect_user_tweets.py", line 45, in
result_type="recent")
File ".../venv/lib/python3.5/site-packages/twitter/api.py", line 424, in GetSearch
data = self._ParseAndCheckTwitter(resp.content.decode('utf-8'))
File ".../venv/lib/python3.5/site-packages/twitter/api.py", line 4688, in _ParseAndCheckTwitter
self._CheckForTwitterError(data)
File ".../venv/lib/python3.5/site-packages/twitter/api.py", line 4715, in _CheckForTwitterError
raise TwitterError(data['errors'])
twitter.error.TwitterError: [{'message': 'Rate limit exceeded', 'code': 88}]
Any ideas?

[jeremylow]

What version of python-twitter are you using and/or can you test against the master branch?

In the interim, I checked out what the problem and it looks like it's Twitter being inconsistent in the rate limit returned in their URL headers, which may be a temporary problem or may be something that we need to take into account. For instance, for the code:

for i in range(0, 175):
    api.GetSearch(term="primary", count=100)
    print(api.rate_limit.get_limit('/search/tweets'))

I am essentially getting two sets of rate limits:

EndpointRateLimit(limit=180, remaining=134, reset=1465417711)
EndpointRateLimit(limit=180, remaining=25, reset=1465417814)
EndpointRateLimit(limit=180, remaining=133, reset=1465417711)
EndpointRateLimit(limit=180, remaining=24, reset=1465417814)
EndpointRateLimit(limit=180, remaining=23, reset=1465417814)
EndpointRateLimit(limit=180, remaining=22, reset=1465417814)
EndpointRateLimit(limit=180, remaining=21, reset=1465417814)
EndpointRateLimit(limit=180, remaining=132, reset=1465417711)
EndpointRateLimit(limit=180, remaining=20, reset=1465417814)
EndpointRateLimit(limit=180, remaining=19, reset=1465417814)

Where one (the lower one) is the real rate limit and the higher one is maybe cached on their servers or whatever server is counting their rate limits is being inconsistently load balanced? I'm not sure.

What I'd suggest you do in the meantime (since there's nothing we can do about this right now), is something like:

for user in users:
    results = api.GetSearch(
        term="from:"+user,
        since_id=sincetweet,
        max_id=maxtweet,
        count=10,
        result_type="recent")
    time.sleep(5.05)

The time.sleep(5.05) call will ensure that you never go over the rate limit for the GetSearch endpoint (15 minute window (900 seconds) divided by 180 allowed calls + a small amount of padding).

[jeremylow]

It's worth noting that Twitter's having issues serving images today as well, so it might be temporary. Either way, you can enforce the correct sleeping behavior manually.

[ctlewitt]

I downloaded the newest version about a week ago. 3-something, I believe?
Thanks for the time.sleep(5.05) idea. I'd been handling it by just having a try-except block that went to sleep for 16 minutes and then repeated the request again before continuing. It was working, but the wait time was frustratingly long when it would follow a Python-Twitter sleep period. It was probably excessive. I'll try your idea instead. Thanks!

[jeremylow]

(I only asked since I noticed that the line numbers were different from master so I wanted to make sure it wasn't something super old; so that's good.)

Yeah, I hope that's a reasonable solution for the moment. It's one of those things where we rely on the information coming back from Twitter, so I'm not really sure how we want to account for inconsistencies in that data, but, generally speaking, if you are having rate limit error issues, even when using the built in Api.sleep_on_rate_limit, I'd suggest manually timing out your calls. The rate limit chart has most endpoints here: https://dev.twitter.com/rest/public/rate-limits and anything else that isn't listed should be 15 calls per 15 minutes.

I'll check out the code above later this week and see if it has fixed itself (if it seems like it has, I'll close the issue). If you run into any further problems, feel free to reopen the issue or file another one.

Thanks!

[ctlewitt]

Thanks!

On Wed, Jun 8, 2016 at 10:02 PM, Jeremy Low notifications@github.com
wrote:

(I only asked since I noticed that the line numbers were different from
master so I wanted to make sure it wasn't something super old; so that's
good.)

Yeah, I hope that's a reasonable solution for the moment. It's one of
those things where we rely on the information coming back from Twitter, so
I'm not really sure how we want to account for inconsistencies in that
data, but, generally speaking, if you are having rate limit error issues,
even when using the built in Api.sleep_on_rate_limit, I'd suggest
manually timing out your calls. The rate limit chart has most endpoints
here: https://dev.twitter.com/rest/public/rate-limits and anything else
that isn't listed should be 15 calls per 15 minutes.

I'll check out the code above later this week and see if it has fixed
itself (if it seems like it has, I'll close the issue). If you run into any
further problems, feel free to reopen the issue or file another one.

Thanks!

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#350 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AHtXugoC_Q5Cmmwa6jlSbOkWnqz27MpMks5qJ3QxgaJpZM4IxCje
.

[kclock]

I changed def _RequestUrl(self, url, verb, data=None, json=None): inapi.py for this problem.
if limit.remaining == 0 -> if limit.remaining < 3
This works well in my environment.

[jeremylow]

This seems to be working properly now with the correct rate limit returned in all cases tested, so I'm going to close the issue as it seems like it was a problem with Twitter's servers. Please reopen if you're still having a problem and maybe we can get to the bottom of it.

[edsu]

I have a program that iterates through all available tweet search results and it throws twitter.error.TwitterError: [{'message': 'Rate limit exceeded', 'code': 88}] after sleeping for a period of time, that is presumably to avoid the rate limit error?

api = twitter.Api(
    consumer_key='a',
    consumer_secret='b',
    access_token_key='c',
    access_token_secret='d',
    sleep_on_rate_limit=True
)

last_id = None

while True:
    tweets = api.GetSearch(term='obama', count=100, max_id=last_id)

    if len(tweets) == 0:
        break

    for tweet in tweets:
        print(tweet.text)
        last_id = tweet.id

Shouldn't python-twitter sleep for the correct amount of time so that this exception doesn't happen?

[bear]

We do, if you see @jeremylow 's comment earlier we often see that different edge servers in Twitter's fleet will respond with different rate-limit values which cause the code to not slow down, and then the next cycle of requests hit another server that has a lower limit and you get limited.

He also mentioned a work-around that allows for a less painful experience

[jeremylow]

There may be an issue with rounding. Mind if I just push a small padding value on the sleep call to master, @bear ?

[edsu]

Why not catch the exception if the user is asking python-twitter to handle rate limiting? I mean, I guess I can too, but it kinda defeats the purpose of asking python-twitter to manage it.

[jeremylow]

Ok, so there's a small padding value (t <= 2s) since we were seemingly sleeping for <= 1 too few seconds. @edsu if you could give this a shot, I'd be interested in your results.

As for catching the error, personally I'd prefer managing it this way and reducing the number of times our library is responsible for going over Twitter's limits. (See also #358 for why I'd rather not dictate the user's actions on hitting a rate limit. A try/except clause would remove the user's ability to deal with the error as they see fit.)

[jeremylow]

(I mean, the user's has said that they'd like the API to sleep, so yeah, that concern is probably moot.)