Skip to content

Commit

Permalink
[query] fix backoff code (#13713)
Browse files Browse the repository at this point in the history
CHANGELOG: Fixes #13704, in which Hail could encounter an
IllegalArgumentException if there are too many transient errors.

I need to do the multiplication in 64-bits so that it does not wrap
around to a large negative value. Then I can use `math.min` with the
maxDelayMs to get us back into 32-bits.

I'm just pushing through a bunch of bugs to get Wenhan unblocked today.
  • Loading branch information
danking committed Sep 27, 2023
1 parent db2c534 commit 481cfc2
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions hail/src/main/scala/is/hail/services/package.scala
Expand Up @@ -42,10 +42,10 @@ package object services {
// Based on AWS' recommendations:
// - https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
// - https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/java/com/amazonaws/retry/PredefinedBackoffStrategies.java
val multiplier = 1 << math.min(tries, LOG_2_MAX_MULTIPLIER)
val ceilingForDelayMs = baseDelayMs * multiplier
val multiplier = 1L << math.min(tries, LOG_2_MAX_MULTIPLIER)
val ceilingForDelayMs = math.min(baseDelayMs * multiplier, maxDelayMs).toInt
val proposedDelayMs = ceilingForDelayMs / 2 + Random.nextInt(ceilingForDelayMs / 2 + 1)
return math.min(proposedDelayMs, maxDelayMs)
return proposedDelayMs
}

def sleepBeforTry(
Expand Down

0 comments on commit 481cfc2

Please sign in to comment.