Add a delay before search_all_tweets calls

[jdfoote]

search_all_tweets has a 1 call per second limit. Currently, tweepy quickly makes a few calls, receives a 429, and then waits for nearly 15 minutes. By simply adding a sleep of 1 second per call, you can make 300 calls in that 15 minutes.

It seems like the place to put this may be in the pagination.py file, as the delay is only a problem when making multiple calls? I'd be happy to make a one-line PR, but I'm not sure where the devs would want to put the sleep.

[mshlis]

I want to +1 this. I get 429's with:

    for response in tweepy.Paginator(twitter_call, **search_params):
        data.extend(response.data)

but not with:

    next_token = 1
    while next_token:
        response = twitter_call(**search_params)
        data.extend(response.data)
        
        next_token = response.meta.get('next_token')
        search_params['next_token'] = next_token        
        time.sleep(1)

[Harmon758]

I'm aware of this issue, but I haven't determined the best way to resolve it yet.
Ideally, I'd rather not have to hard-code the rate limit into the library, but that might end up being necessary, as this rate limit isn't in the response headers and seems to only be declared in the endpoint's documentation.
For now, a workaround is simply to sleep 1 second between each request, while iterating through the Paginator responses.

[jdfoote]

That makes sense. Maybe just some documentation, then?

[Harmon758]

Upon consideration, it might not be best to handle this within Tweepy.

There might be users who do processing within the loop that takes up a significant amount of time or even takes longer than a second, so a simple 1 second sleep wouldn't be ideal.

The alternative would be saving the timestamp that the last request was made at and sleeping until a second has passed, but that might end up being exactly a second and there would need to be considerations for adding some jitter.

I think you're right and the simplest way forward right now would be to document it and allow the user to handle it themselves.
If it becomes necessary at some point, handling of this rate limit can always be added later.

PS

@jdfoote I saw your video 👍

Some notes:

• start_time and end_time can be datetime objects if you don't want to have to remember the string format that Twitter's API requires
• Tweepy's API v2 models have a data attribute (still needs to be documented, but it's not going anywhere) that provides the entire data dictionary

[Harmon758]

I've added a FAQ section on this for now.

[xrisk]

@Harmon758 This rate limit should be present in at least the flatten api call I think. There is no way to insert 1 second sleeps there. On a related note, I feel this should be documented on the main docs page for pagination, not in the FAQ.