Skip to content

Rate Limited API How to deal with

Andy Piper edited this page Jun 24, 2019 · 3 revisions

Below is one way of dealing with the RateLimit error - hack the code so you don't make as many requests.

Sebastian pointed out on the list that a better way is to use the new bulk ID methods

https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids.html

And then walk thru that using your code. This is a change that we are chewing on for the next python-twitter update.

A version of GetFollowers() that pulls chunks of results so you don't get rate limited.

  def GetFollowers(self, user_id=None, screen_name=None, cursor=-1, skip_status=False, include_user_entities=False, count=10):

    '''Fetch the sequence of twitter.User instances, one for each follower

    The twitter.Api instance must be authenticated.

    Args:
      user_id:
        The twitter id of the user whose followers you are fetching.
        If not specified, defaults to the authenticated user. [Optional]
      screen_name:
        The twitter name of the user whose followers you are fetching.
        If not specified, defaults to the authenticated user. [Optional]
      cursor:
        Should be set to -1 for the initial call and then is used to
        control what result page Twitter returns [Optional(ish)]
      skip_status:
        If True the statuses will not be returned in the user items.
        [Optional]
      include_user_entities:
        When True, the user entities will be included.

    Returns:
      A sequence of twitter.User instances, one for each follower
    '''
    if not self._oauth_consumer:
      raise TwitterError("twitter.Api instance must be authenticated")
    url = '%s/followers/list.json' % self.base_url
    result = []
    parameters = {}
    if user_id is not None:
      parameters['user_id'] = user_id
    if screen_name is not None:
      parameters['screen_name'] = screen_name
    if skip_status:
      parameters['skip_status'] = True
    if include_user_entities:
      parameters['include_user_entities'] = True
    remaining=count
    ratelimited=False
    while remaining > 1:
      remaining -= 1
      parameters['cursor'] = cursor
      json = self._FetchUrl(url, parameters=parameters)
      try:
        data = self._ParseAndCheckTwitter(json)
        result += [User.NewFromJsonDict(x) for x in data['users']]
        if 'next_cursor' in data:
          if data['next_cursor'] == 0 or data['next_cursor'] == data['previous_cursor']:
            break
          else:
            cursor = data['next_cursor']
        else:
          break
      except TwitterError:
        ratelimited = True
        break
    return (cursor, result, ratelimited)
Clone this wiki locally