Client-side throttling causes wait on request to K8s API

[laughingman7743]

Summary

Hi, I am using ArgoWorkflows on GKE in a single namespace.
When using a workflow template, it seems that the controller sends a request to the K8s API to get the workflow template when the workflow is executed, but when there are many workflows to execute, the following log is output and the workflow is not executed. It seems that the workflow cannot be started because it is trapped by the slot control of the K8s API.

argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:01.059447       1 request.go:601] Waited for 1.02913332s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:11.059477       1 request.go:601] Waited for 11.0120979s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:21.109618       1 request.go:601] Waited for 21.00025731s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:31.159278       1 request.go:601] Waited for 25.743405682s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:41.208941       1 request.go:601] Waited for 25.741356153s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:00:51.209704       1 request.go:601] Waited for 25.741993094s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:01:01.209713       1 request.go:601] Waited for 25.741880557s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:01:21.260237       1 request.go:601] Waited for 25.791448291s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template
argo-workflows-workflow-controller-64f8db5cc8-dgssc controller I0510 15:01:31.308984       1 request.go:601] Waited for 25.787429904s due to client-side throttling, not priority and fairness, request: GET:https://10.32.128.1:443/apis/argoproj.io/v1alpha1/namespaces/my-namespace/workflowtemplates/my-workflow-template

I am encountering cases, especially with Cron workflows that regularly run hundreds of jobs at the same time, where these logs are output and it takes about 10-15 minutes to start the Cron workflow. (In some cases it can take 30 minutes or more to start). Is it possible to reduce the number of requests to the K8s API?

The K8s API client seems to have a cache mechanism, can we improve it to make better use of the cache to reduce API requests?
https://github.com/kubernetes/sample-controller/blob/v0.25.0/docs/controller-client-go.md
It seems that controller-runtime has a DelegatingClient that uses the cache for read processes such as Get and List.
https://github.com/kubernetes-sigs/controller-runtime/blob/v0.14.6/pkg/client/split.go#L39-L44

A similar issue is registered on ArgoCD.
argoproj/argo-cd#8620

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[laughingman7743]

https://argoproj.github.io/argo-workflows/running-at-massive-scale/#overwhelmed-kubernetes-api

[vadasambar]

@laughingman7743 why was this issue closed? I don't think this problem is due to load on K8s API.
Related:

• client-go: do not print the request latency logs at level 0 by default kubernetes/kubernetes#101634 (comment)
• kubectl: discovery is throttled when there are lots of resources (CRDs) kubernetes/kubernetes#105489
• https://jonnylangefeld.com/blog/the-kubernetes-discovery-cache-blessing-and-curse

[laughingman7743]

I closed this issue because I found the following section in the documentation
#11069 (comment)
I tried changing the settings as per the documentation, but it did not solve the issue at all. 😩
I have now changed it to not use workflow templates to avoid this issue.

Maybe the following issue is related to this issue. 🤔
#11372
If necessary, you should be good to register this issue anew.