Add req_chunk()
#307
Add req_chunk()
#307
Conversation
|
@hadley It would also make sense to add support for |
R/chunk_req_perform.R
Outdated
| #' n <- length(requests) | ||
| #' paths <- character(n) | ||
| #' responses <- list() | ||
| #' for (i in seq2(1, n)) { | ||
| #' paths[[i]] <- tempfile() | ||
| #' responses[[i]] <- req_perform(requests[[i]], path = paths[[i]]) | ||
| #' } | ||
| #' } |
There was a problem hiding this comment.
It's interesting to think about using multi_req_perform() here.
#Conflicts: # R/utils.R
|
@hadley Now |
| #' | ||
| #' responses <- req_perform_multi(req) | ||
| #' } | ||
| req_chunk <- function(req, |
There was a problem hiding this comment.
It might be nice to have a helper to create a list of all requests so that one can perform them in parallel.
This would also work for two pagination patterns (pagination via offset and via page index). But this should be handled in a separate PR.
| }, | ||
| cancel_on_error = cancel_on_error, | ||
| error_message = "When parsing response {i}.", | ||
| error_class = "httr2_parse_failure", |
There was a problem hiding this comment.
Or would you prefer to have the same class when the request fails and when the parsing fails? I.e. <httr2_failure> in both cases.
| out <- out[seq2(1, i)] | ||
| } | ||
|
|
||
| vctrs::list_unchop(out) |
There was a problem hiding this comment.
This doesn't work well in case cancel_on_error = FALSE and there actually is an error. Should we only unchop if cancel_on_error = TRUE?
There was a problem hiding this comment.
Or only unchop the results corresponding to requests that succeeded?
| #' ) | ||
| #' | ||
| #' responses <- req_perform_multi(req_pokemon) | ||
| req_perform_multi <- function(req, |
There was a problem hiding this comment.
We need to think about the naming of functions. There currently is multi_req_perform() to perform a list of requests in parallel. It might make sense to rename this to something like reqs_perform_parallel() (though I'm not quite sure we want to add the prefix reqs).
| @@ -0,0 +1,100 @@ | |||
| #' Chunk a request | |||
There was a problem hiding this comment.
Or should this be Batch a request and the function name req_batch()?
There was a problem hiding this comment.
Yeah, I like batch better. req_ is probably not quite the right prefix (since it doesn't return a request) but I don't see a better alternative.
#Conflicts: # R/paginate.R # man/paginate_req_perform.Rd
#Conflicts: # NEWS.md # R/paginate.R # man/paginate_req_perform.Rd # man/req_paginate.Rd
| #' If you need more control use a combination of [req_perform()] and | ||
| #' [paginate_next_request()] to iterate through the pages yourself. | ||
| #' | ||
| #' [req_next_multi()] to iterate through the pages yourself. |
There was a problem hiding this comment.
Or should this be resp_next_request()?
| @@ -0,0 +1,100 @@ | |||
| #' Chunk a request | |||
There was a problem hiding this comment.
Yeah, I like batch better. req_ is probably not quite the right prefix (since it doesn't return a request) but I don't see a better alternative.
| out <- out[seq2(1, i)] | ||
| } | ||
|
|
||
| vctrs::list_unchop(out) |
There was a problem hiding this comment.
Or only unchop the results corresponding to requests that succeeded?
| tryCatch( | ||
| f(...), | ||
| error = function(cnd) { | ||
| cli::cli_abort( |
|
Superseded by #353 |
Closes #303.