Provide more user-friendly documentation for API rate limit

[donghaichuan]

My cluster version：
system：Ubuntu 20.04.1 LTS
kernel：5.10.3-051003-generic
k8s：v1.18.4
docker：19.3.5
cilium：1.9.1

I deployed a k8s cluster with the cilium network plugin , there is 3000+ pods and 30 nodes in my cluster, I add one new node into cluster ,I found many error logs on new node ,as follows：

Also the status of the pods scheduled to the new node are ContainerCreating，as follows：

After a few minutes：

I watch new node's cilium agent logs，as follows：

Please help me to solve this problem，thanks very much。

[pchaigno]

Could you please add a sysdump or include those as text instead of screenshot? It's hard to parse information from screenshots...

[donghaichuan]

The new node error syslog file details as follows:

Mar 19 12:24:55 k8sdeploy-node63 kubelet[1608]: E0319 12:24:55.651067 1608 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "a89f1f10c0655325f909ba937ce1b5d8db70fbc0a9fa67daed62dca3de5a8f73" network for pod "es-white-list-cronjob-1616127480-nbczk": networkPlugin cni failed to set up pod "es-white-list-cronjob-1616127480-nbczk_caibin1" network: Unable to create endpoint: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 429): {}
Mar 19 12:24:55 k8sdeploy-node63 kubelet[1608]: E0319 12:24:55.651153 1608 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "es-white-list-cronjob-1616127480-nbczk_caibin1(4dc4b788-66a2-4c25-8e55-f3994cf474f4)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "a89f1f10c0655325f909ba937ce1b5d8db70fbc0a9fa67daed62dca3de5a8f73" network for pod "es-white-list-cronjob-1616127480-nbczk": networkPlugin cni failed to set up pod "es-white-list-cronjob-1616127480-nbczk_caibin1" network: Unable to create endpoint: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 429): {}
Mar 19 12:24:55 k8sdeploy-node63 kubelet[1608]: E0319 12:24:55.651186 1608 kuberuntime_manager.go:727] createPodSandbox for pod "es-white-list-cronjob-1616127480-nbczk_caibin1(4dc4b788-66a2-4c25-8e55-f3994cf474f4)" failed: rpc error: code = Unknown desc = failed to set up sandbox container "a89f1f10c0655325f909ba937ce1b5d8db70fbc0a9fa67daed62dca3de5a8f73" network for pod "es-white-list-cronjob-1616127480-nbczk": networkPlugin cni failed to set up pod "es-white-list-cronjob-1616127480-nbczk_caibin1" network: Unable to create endpoint: response status code does not match any response statuses defined for this endpoint in the swagger spec (status 429): {}

The new node's cilium agent error log details as follows:

level=warning msg="Not processing API request. Wait duration for maximum parallel requests exceeds maximum" error="context deadline exceeded" maxWaitDuration=15s name=endpoint-create parallelRequests=3 subsys=rate uuid=efabe2d6-886a-11eb-832b-3cecef001f42
level=info msg="API call has been processed" error="timed out while waiting to be served with 3 parallel requests: context deadline exceeded" name=endpoint-create processingDuration=0s subsys=rate totalDuration=15.000634348s uuid=efabe2d6-886a-11eb-832b-3cecef001f42 waitDurationTotal=0s
level=info msg="API call has been processed" error="request would have to wait 18.736302926s to be served (maximum wait duration: 622.727216ms)" name=endpoint-create processingDuration=0s subsys=rate totalDuration=16.396271422s uuid=ee181d17-886a-11eb-832b-3cecef001f42 waitDurationTotal=14.377272784s
level=info msg="API call has been processed" error="request would have to wait 16.717671024s to be served (maximum wait duration: 18.52909ms)" name=endpoint-create processingDuration=0s subsys=rate totalDuration=17.000479009s uuid=edbbea7f-886a-11eb-832b-3cecef001f42 waitDurationTotal=14.98147091s
level=info msg="Processing API request with rate limiter" maxWaitDuration=15s name=endpoint-create parallelRequests=3 subsys=rate uuid=ee181d17-886a-11eb-832b-3cecef001f42
level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager

@pchaigno If these logs cannot determine the cause of the problem， Please tell me your email，I can send log file to you，Thanks very much。

[pchaigno]

You can attach a sysdump to any GitHub comment.

[joestringer]

status 429 means "Too Many Requests". Basically, the scheduling rate is too high, so to avoid consuming too much resources either on the local node or DoS'ing the kube-apiserver, Cilium has a ratelimiter in place for local API requests to establish network connectivity for newly created pods. These limits are dynamically adjusted based on the ability of Cilium to handle incoming requests, but you can also override the automatic rate.

These limits can be tweaked by following the documentation here if you believe the limits are too low:
https://docs.cilium.io/en/stable/configuration/api-rate-limiting/

If you're able to raise the limits and have success with deploying at a higher rate, I would be curious to hear about the limits that you use and what you were successful with. This will also depend on the CPU resources available on your individual nodes and the pod deployment churn rate (how many pods you deploy per second on a given node).

[donghaichuan]

My all physical nodes are configured with 192G memory and 80 core CPU，so I think the problem is not caused by configuration.

I have read the api-rate-limiting offical documentation of cilium ，because I‘m not a developer，I still don’t know how to modify the rate limit of cilium. I installed cilium by helm. I don't know where can set parameter of rate limit.Can you tell me some specific methods of operation.

My cluster will add 30 new nodes in future, this problem prevents me to add new nodes,please help me ...

[donghaichuan]

You can attach a sysdump to any GitHub comment.
Is the log content I provided too little? I only provide content that recurs more frequently.

[donghaichuan]

I tried to update cilium's configmap file, add (api-rate-limit: "endpoint-create=rate-limit:64/s,rate-burst:64"),and restart the cilium agent pod, I found new node's logs were still have (status 429),all pod are creating.

I also tried to set other values,for example,(rate-limit:4/s,rate-burst:8),(rate-limit:8/s,rate-burst:16),(rate-limit:16/s,rate-burst:32),These have no obvious effect.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.