Skip to content

Commit

Permalink
Increase the max wait time for EndpointCreate API requests
Browse files Browse the repository at this point in the history
In the event that a node has a burst of PodSanboxCreate requests, API
requests will pile up. However, we should wait longer for the queue to
clear before short-circuting and returing failure.

This is because the kubelet has a relatively relaxed timeout for
PodSandbox creation -- 4 minutes. Furthermore, if we return a failure
here, it is propagated all the way back through containerd to kubelet,
which will tear down the entire PodSandbox and try again, which can be
expensive.

So, increase the maximum queue duration time to 1 minute. That should
hopefully give enough time for the queue to clear.

Signed-off-by: Casey Callendrello <cdc@isovalent.com>
  • Loading branch information
squeed authored and borkmann committed Jun 20, 2023
1 parent e956bb1 commit c8eeebe
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion daemon/cmd/api_limits.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ var apiRateLimitDefaults = map[string]rate.APILimiterParameters{
RateLimit: 0.5,
RateBurst: 4,
ParallelRequests: 4,
MinParallelRequests: 2,
SkipInitial: 4,
MaxWaitDuration: 15 * time.Second,
MaxWaitDuration: 60 * time.Second, // Kubelet has a PodSandbox creation timeout of 4 minutes, in total.
Log: false,
},
// DELETE /endpoint/{id}
Expand Down

0 comments on commit c8eeebe

Please sign in to comment.