Skip to content

Commit

Permalink
Increase the max wait time for EndpointCreate API requests
Browse files Browse the repository at this point in the history
In the event that a node has a burst of PodSanboxCreate requests, API
requests will pile up. However, we should wait longer for the queue to
clear before short-circuting and returing failure.

This is because the kubelet has a relatively relaxed timeout for
PodSandbox creation -- 4 minutes. Furthermore, if we return a failure
here, it is propagated all the way back through containerd to kubelet,
which will tear down the entire PodSandbox and try again, which can be
expensive.

So, increase the maximum queue duration time to 1 minute. That should
hopefully give enough time for the queue to clear.

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
  • Loading branch information
squeed committed Jun 19, 2023
1 parent 9813a0b commit 4ff1ef8
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion daemon/cmd/api_limits.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ var apiRateLimitDefaults = map[string]rate.APILimiterParameters{
RateLimit: 0.5,
RateBurst: 4,
ParallelRequests: 4,
MinParallelRequests: 2,
SkipInitial: 4,
MaxWaitDuration: 15 * time.Second,
MaxWaitDuration: 60 * time.Second, // Kubelet has a PodSandbox creation timeout of 4 minutes, in total.
Log: false,
},
// DELETE /endpoint/{id}
Expand Down

0 comments on commit 4ff1ef8

Please sign in to comment.