Configure robust retries and connection timeouts for http commands like fetch, post

[Veetaha]

Related problem

Network access is always flaky. Sometimes there may be some network glitch, or some server may feel unhealthy, or many clients may connect to the server simultaneously causing big load. So things like downloading content from some host or posting some mutation may fail. For this reason, commands like fetch/post (are there other http commands?) should include configurations of retries and separate connection timeouts of the network operations.

Describe the solution you'd like

There should be two retry algorithms implemented: a fixed delay retry and exponential backoff with jitter.

The fixed delay retry is simple. The user specifies the duration between each retries and the number of maximum retries to perform. We could also optionally support jitter for this algorithm to add some randomness to the fixed delay, but I am not sure that existing mature networking tools provide jitter for fixed-delay retries, so maybe we don't need that 🤔.

The exponential backoff with jitter is best described in an article by AWS, even though the text there begins with talking about optimistic concurrency control (OCC) with DynamoDB the rest of the article is pretty generic. The article shows that the best option in the client-server OCC-model communication is using exponential backoff with jitter. The jitter part is very important to avoid the thundering herd problem.

To be more specific, there should be CLI parameters to configure the retry and timeouts behavior of fetch.

We could take a look and maybe even implement the same configurations that curl provides.

Here is a good description of the retries config in curl: link to the article.
Here is how these parameters appear in curl's help output:

     --retry <num>   Retry request if transient problems occur
     --retry-connrefused Retry on connection refused (use with --retry)
     --retry-delay <seconds> Wait time between retries
     --retry-max-time <seconds> Retry only within this period

Here is a good description of the timeouts config in curl: link to the article
Here is how these parameters appear in curl's help output:

     --connect-timeout <fractional seconds> Maximum time allowed for connection
 -m, --max-time <fractional seconds> Maximum time allowed for transfer

[rgwood]

Hi, I agree that the commands should retry.

There should be two retry algorithms

Is there a particular reason you want 2? I think we'd be fine with 1 algorithm (exponential backoff probably) and that would keep the number of flags down.

[fdncred]

retries would be a great addition but I think we also should be careful about adding too many command line parameters as well.

[Veetaha]

Fixed retries aren't that important. In fact, we may support only exponential backoff initially. I was thinking about fixed retry algorithm, because curl supports it, and because I used it in Rust when implementing a polling loop that waits until a server is available.
So, let's begin with exponential only ✔️

[kgadek]

How about implementing this as custom command (perhaps built-in?), like this?

source
| …
| exponential-retry
    --max-retries=10
    --cap=10min
    --start=2sec
    --multiplier=2
    --jitter=5sec
  { |it|
      # whatever commands here will be retried
  }
| …
| sink

This way, we could keep http simpler while also providing exponential backoff for other commands (like invocations of AWS CLI).

---

I've been able to implement these things, but this until is kinda ugly (see #11453).

def until [cond: closure] {
    let piped = $in
    mut index = 0
    let input_len = $piped | length
    mut result = null
    # ^-- instead there should be something like yield/next
    # from Python to tell if there's next element to be taken
    
    while ($result | is-empty) and ($index < $input_len) {
        let index_const = $index
        let result_const = do $cond ($piped | get $index_const)
        # ^-- here we need `$piped`, because `$in` results in "variable not found"
        $result = $result_const
        $index += 1
    }
    if ($result | is-empty) {
        error make {msg: "oops"}
    } else {
        $result
    }
}

def retry [
  --retries (-r): int,
  mkDelay: closure,
  op: closure
] {
    let piped = $in
    0..<$retries | until {|attempt_index|
        let res = do $op
        if not ($res | is-empty) {
            $res
        } else {
            let sleep_time: duration = do $mkDelay $attempt_index
            print $"Will sleep ($sleep_time)"
            sleep $sleep_time
        }
    }
}