lookup_users for more than 18,000 screen names

[rayms]

Is there a way to use the lookup_users function for more the max 18,000 screen names? The documentation states that the max response is 18,000 per token. Does this mean there is no way to request the data for more than 18,000 users?

[mkearney]

@rayms Thanks for pointing out the documentation number. The max number of users should be 90,000 per every 15 minutes. I've updated the documentation to be more accurate (and to make note that it's possible to iterate through more than the max...it just requires navigating rate limit resets). Eventually, this function will hopefully include a retryonratelimit argument to automate sleeping between rate limit resets, but I probably won't get to that until the next CRAN release (I hope to have a new CRAN release very soon! So the one after that.).

[rayms]

hi @mkearney , sorry to bump this - I wondered if you could advise on the code for iterating through the max number of 90,000 users? I am not sure this is doing the trick, as it seems to retrieve the first 90,000 very, very slowly. I adapted it from a previous answer you gave on the get_timelines function for multiple users. Thanks a lot!

data <- vector("list", length(sample))

for (i in seq_along(data)) {
  data[[i]] <-
    lookup_users(sample[i], parse = TRUE)
  if (i %% 90000L == 0L) {
    rl <- rate_limit("users/lookup")
    Sys.sleep(as.numeric(rl$reset, "secs"))
  }
}

[johnatasjmo]

@rayms , the above code doesnt work well, rate_limit goes over the limit.

@mkearney, any estimated timeline to get retryonratelimit included? or how can we improve the previous lines of code to make it work? Thanks!

[neilcharles]

I found this issue while Googling for a lazy way around the 90,000 limit. My solution below, which might not be the most elegant but it works for me.

lookup_many_users <- function(users, twitter_token, retry_limit = 5){
  require(rtweet)
  
  breaks <- seq(1, length(users), 89999)
  
  if(breaks[length(breaks)] != length(users)){
    breaks <- c(breaks, length(users))
  }
  
  user_details <- NULL
  
  for(i in 1:(length(breaks) -1)){
    
    attempt <- 0
    
    while(is.null(user_details) && attempt <= retry_limit){
      
      attempt <- attempt + 1
      
      try({
        user_details <- lookup_users(users[breaks[i]:breaks[i+1]], token = twitter_token)

        Sys.sleep(15 * 60) #wait 15 minutes for rate limit to reset before proceeding
      })
    }
    
    if(is.null(user_details)){
      stop("failed to get users")
    }    
    
    if(i == 1){
      all_user_details <- user_details
    } else {
      all_user_details <- rbind(all_user_details, user_details)
    }
    
    user_details <- NULL
  }
  
  return(all_user_details)
  
}

[lccastrop]

I found this issue while Googling for a lazy way around the 90,000 limit. My solution below, which might not be the most elegant but it works for me.

lookup_many_users <- function(users, twitter_token, retry_limit = 5){
  require(rtweet)
  
  breaks <- seq(1, length(users), 89999)
  
  if(breaks[length(breaks)] != length(users)){
    breaks <- c(breaks, length(users))
  }
  
  user_details <- NULL
  
  for(i in 1:(length(breaks) -1)){
    
    attempt <- 0
    
    while(is.null(user_details) && attempt <= retry_limit){
      
      attempt <- attempt + 1
      
      try({
        user_details <- lookup_users(users[breaks[i]:breaks[i+1]], token = twitter_token)

        Sys.sleep(15 * 60) #wait 15 minutes for rate limit to reset before proceeding
      })
    }
    
    if(is.null(user_details)){
      stop("failed to get users")
    }    
    
    if(i == 1){
      all_user_details <- user_details
    } else {
      all_user_details <- rbind(all_user_details, user_details)
    }
    
    user_details <- NULL
  }
  
  return(all_user_details)
  
}

friend, what do I send on Twitter_token? Can you give an example?
infoSeguidoresMios <- lookup_many_users(seguidoresMios$user_id, get_token())

it does not work