Extract query parameters

[mgirlich]

I spend a few minutes to try out httr2. Great work with the new oauth system. I didn't really managed to use oauth with httr but it only took me two minutes to make it work in httr2 😄

I want to extract the url query paramters of a request and wondered whether there is an easier way than the following

library(httr2)
req <- request("https://example.com/path/to/page?name=ferret&color=purple")

req_queries <- url_parse(req$url)$query
req_queries$name
#> [1] "ferret"

^{Created on 2021-07-29 by the reprex package (v2.0.0)}

[hadley]

Glad you're enjoying it 😄 I'm now attempting to wrap it up and get at least an initial version on CRAN in the next week or two.

Do you mind explaining a bit more of the context here? I'm generally expect that you're the one modifying the request, so I've mostly only provided getters, not setters.

[mgirlich]

I have an api with pagination and wanted to follow your next_page() example. The api has (at least) two different versions for pagination 🙄 : one with a pagination cursor in the response body and one with a start index and a page size.
My code basically looks like this

api_req_paginate <- function(req, f_next_page, n_pages = NULL, page_size = 5000,
                             spec = NULL, path = NULL) {
  out <- vector("list", n_pages %||% 100)
  i <- 1L

  repeat({
    out[[i]] <- httr2::req_perform(req)
    if (!is.null(n_pages) && i == n_pages) {
      break
    }

    req <- f_next_page(req, out[[i]])
    if (is.null(req)) {
      break
    }

    i <- i + 1L
    if (i > length(out)) {
      length(out) <- length(out) * 2L
    }
  })

  if (i != length(out)) {
    out <- out[seq_len(i)]
  }

  purrr::map_dfr(out, api_parse_response, spec = spec, path = path)
}

api_req_paginate_startindex <- function(req, n_pages = NULL, page_size = 5000,
                                        spec = NULL, path = NULL) {
  api_req_paginate(
    req,
    f_next_page = function(req, resp) api_next_page_index(req, resp, page_size = page_size),
    n_pages = n_pages,
    spec = spec,
    path = path
  )
}

api_next_page_index <- function(req, resp, page_size) {
  if (length(httr2::resp_body_json(resp)) < page_size) {
    return(NULL)
  }

  resp_query <- httr2::url_parse(resp$url)$query
  startIndex <- as.integer(resp_query$startIndex %||% 0L) + page_size
  httr2::req_url_query(req, startIndex = startIndex, count = page_size)
}

api_req_paginate_cursor <- function(req, n_pages = NULL, spec = NULL, path = NULL) {
  api_req_paginate(
    req,
    f_next_page = api_next_page_cursor,
    n_pages = n_pages,
    spec = spec,
    path = path
  )
}

api_next_page_cursor <- function(req, resp) {
  resp_parsed <- httr2::resp_body_json(resp, check_type = FALSE)
  cursor <- resp_parsed$nextCursor
  if (is.null(cursor)) {
    return(NULL)
  }

  data_new <- jsontools::json_mutate(req$body$data, .cursor = cursor)
  req %>%
    httr2::req_body_raw(
      data_new,
      type = req$body$type
    )
}

[hadley]

Hmmm, for that style of pagination, I don't think I'd use an iterative approach because you can generate all the requests up front, and then do them in parallel. OTOH I can see that it would be nice to be able to use the same API in both cases.

[hadley]

Also I just wanted to use resp_url() in a test, so it would definitely be handy to have a few more helpers here.