New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need function for retrying a request #353

Closed
hadley opened this Issue Apr 2, 2016 · 2 comments

Comments

Projects
None yet
1 participant
@hadley
Member

hadley commented Apr 2, 2016

  • Fixed number of tries
  • Exponential back off
  • Built-in randomness
@hadley

This comment has been minimized.

Member

hadley commented May 20, 2016

@craigcitro if you're doing PRs, any interest in doing one for this?

@hadley

This comment has been minimized.

Member

hadley commented May 22, 2016

Here's a first stab at it:

#' @inheritParams GET
#' @inheritParams POST
#' @param times Number of times to retry
#' @param pause_base,pause_cap This method uses exponential back-off of with
#'   full jitter (\url{}) - this means that each request will randomly wait
#'   between 0 and \code{pause_base * 2 ^ attempt} seconds, up to a maximum of 
#'   \code{pause_cap} seconds.
retry <- function(verb, url = NULL, config = list(), ...,
                  body = NULL, encode = c("multipart", "form", "json", "raw"),
                  times = 3, pause_base = 1, pause_cap = 60,
                  handle = NULL) {
  stopifnot(is.numeric(times), length(times) == 1L)

  hu <- handle_url(handle, url, ...)
  req <- request_build(verb, hu$url, body_config(body, match.arg(encode)), config, ...)
  resp <- request_perform(req, hu$handle$handle)

  i <- 1
  while (i <= .times && http_error(resp)) {
    resp <- request_perform(req, hu$handle$handle)

    backoff_full_jitter(i, pause_base, pause_cap)
    i <- i + 1
  }

  resp
}

backoff_full_jitter <- function(i, pause_base = 1, pause_cap = 60, quiet = FALSE) {
  stopifnot(is.integer(pause_base), length(pause_base) == 1L)
  stopifnot(is.integer(pause_cap), length(pause_cap) == 1L)

  length <- ceiling(runif(min(pause_cap, pause_base * 2 ^ i)))
  if (!quiet) {
    message("Retrying in ", length, " seconds")
  }
  Sys.sleep(length)
}

I'm using the technique from https://www.awsarchitectureblog.com/2015/03/backoff.html.

@craigcitro do you have any thoughts on the base and cap pauses?

hadley added a commit that referenced this issue May 24, 2016

Implement RETRY()
This keeps retrying until a request succeeds (in a server friendly way)

Fixes #353

hadley added a commit that referenced this issue May 25, 2016

Implement RETRY()
This keeps retrying until a request succeeds (in a server friendly way)

Fixes #353

@hadley hadley closed this in #372 May 31, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment