Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need function for retrying a request #353

Closed
hadley opened this issue Apr 2, 2016 · 2 comments
Closed

Need function for retrying a request #353

hadley opened this issue Apr 2, 2016 · 2 comments

Comments

@hadley
Copy link
Member

hadley commented Apr 2, 2016

  • Fixed number of tries
  • Exponential back off
  • Built-in randomness
@hadley
Copy link
Member Author

hadley commented May 20, 2016

@craigcitro if you're doing PRs, any interest in doing one for this?

@hadley
Copy link
Member Author

hadley commented May 22, 2016

Here's a first stab at it:

#' @inheritParams GET
#' @inheritParams POST
#' @param times Number of times to retry
#' @param pause_base,pause_cap This method uses exponential back-off of with
#'   full jitter (\url{}) - this means that each request will randomly wait
#'   between 0 and \code{pause_base * 2 ^ attempt} seconds, up to a maximum of 
#'   \code{pause_cap} seconds.
retry <- function(verb, url = NULL, config = list(), ...,
                  body = NULL, encode = c("multipart", "form", "json", "raw"),
                  times = 3, pause_base = 1, pause_cap = 60,
                  handle = NULL) {
  stopifnot(is.numeric(times), length(times) == 1L)

  hu <- handle_url(handle, url, ...)
  req <- request_build(verb, hu$url, body_config(body, match.arg(encode)), config, ...)
  resp <- request_perform(req, hu$handle$handle)

  i <- 1
  while (i <= .times && http_error(resp)) {
    resp <- request_perform(req, hu$handle$handle)

    backoff_full_jitter(i, pause_base, pause_cap)
    i <- i + 1
  }

  resp
}

backoff_full_jitter <- function(i, pause_base = 1, pause_cap = 60, quiet = FALSE) {
  stopifnot(is.integer(pause_base), length(pause_base) == 1L)
  stopifnot(is.integer(pause_cap), length(pause_cap) == 1L)

  length <- ceiling(runif(min(pause_cap, pause_base * 2 ^ i)))
  if (!quiet) {
    message("Retrying in ", length, " seconds")
  }
  Sys.sleep(length)
}

I'm using the technique from https://www.awsarchitectureblog.com/2015/03/backoff.html.

@craigcitro do you have any thoughts on the base and cap pauses?

hadley added a commit that referenced this issue May 24, 2016
This keeps retrying until a request succeeds (in a server friendly way)

Fixes #353
hadley added a commit that referenced this issue May 25, 2016
This keeps retrying until a request succeeds (in a server friendly way)

Fixes #353
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant