Significantly reduce retry duration of service discovery #1541
Conversation
The current 66s is quite long and means you don't see your (generally permanent) error quickly enough
There was a problem hiding this comment.
Thanks for creating this PR @jackkleeman. I like this PR more than #1540 since 66s is probably a bit too long. The one thing we should make sure is that the exponential retry policy can work with the all deployments (in terms of response time) we want Restate to work with (maybe the expected response time * 2 being smaller than the max pause between the second to last and last attempt or so). Do you have an idea how long it can take a cold lambda to respond?
| // Total duration roughly 1s | ||
| let retry_policy = RetryPolicy::exponential(Duration::from_millis(100), 2.0, Some(4), None); |
There was a problem hiding this comment.
How long does it take to spin up a cold Lambda?
There was a problem hiding this comment.
this duration is unrelated to the timeout on the request to the lambda - its just the duration between retries. in the lambda case the first request will block on the cold start, and then likely succeed. if it somehow fails transiently, an immediate subsequent retry will most likely not see a cold start, and then succeed immediately. in no scenario would a super slow cold start lead us to breach this retry policy
There was a problem hiding this comment.
True, it is the duration between failures. Thanks for the clarification.
The current 66s is quite long and means you don't see your (generally permanent) error quickly enough