Pagination of results

[schochastics]

A workaround is explained in the wiki but we need to built this in the package

[schochastics]

get_account_following <- function(id,max_id,since_id,limit = 40, token = NULL, parse = TRUE){
  path <- paste0("api/v1/accounts/",id,"/following")
  n <- limit   #CHANGE
  params <- list(limit = min(limit,40))    #CHANGE
  if (!missing(max_id)) {
    params$max_id <- max_id
  }
  if (!missing(since_id)) {
    params$since_id <- since_id
  }
  
  process_request1(token = token,path = path,
                  params = params,
                  parse = parse, FUN = v(parse_account),n = n,page_size = 40L) #CHANGE
}

process_request <- function(token = NULL,
                            path,
                            instance = NULL,
                            params,
                            anonymous = FALSE,
                            parse = TRUE,
                            FUN = identity,
                            n = 1,page_size=40L #CHANGE
){
 #CHANGE FROM
  pages <- ceiling(n/page_size)
  output <- vector("list")
  for(i in seq_len(pages)){
    tmp <- make_get_request(token = token,path = path,
                               instance = instance, params = params,
                               anonymous = anonymous)  
    output <- c(output,tmp)
    attr(output,"headers") <- attr(tmp,"headers")
    if(is.null(attr(tmp,"headers")[["max_id"]])){
      break
    }
    params[["max_id"]] <- attr(tmp,"headers")[["max_id"]]
  }
  #CHANGE TO
  if (isTRUE(parse)) {
    header <- attr(output,"headers")
    
    output <- FUN(output)
    attr(output,"headers") <- header
  }
  return(output)
}

@chainsawriot WIP of the pagination. Just to get some feedback (see #CHANGE). I try to not make the process visible at all for the user. New internal parameters would be n (the actual number of records to return) and page_size (the maximal records returned for an endpoint).
Few things:

• rate limit checking is easy to add. Already have a wait function implemented.
• So far only works with "max_id" (i.e. going backward). But I have a plan for since_id
• Doesnt return the exact n, but multiples of the page_size. I think this is ok so we do not throw away fetched results (rtweet does that too)

What do you think?

[chainsawriot]

@schochastics As said in the httr's Best Practices, there are several ways to deal with this.

But from my view, it sounds okay. I would keep the current default. Mainly because I am lazy to create all the test cases again. Please try to make it works without breaking the current test cases.

[schochastics]

Yes not breaking tests is my main goal besides not changing any defaults. I will not make in PR until I know that nothing breaks. Thanks for the feedback