API request client-side throttling

[josegarridoa]

What happened?

After installing and running provider-aws alongside Crossplane the API start throttling and slowing down the requests
and showing:

I1210 12:45:57.240716   71637 request.go:665] Waited for 1.196258171s due to client-side throttling, not priority and fairness, request: GET:https://api.k8s.<redacted>.io/apis/cloudwatchlogs.aws.crossplane.io/v1alpha1?timeout=32s

How can we reproduce it?

It happens by just installing provider-aws in a cluster. It immediately goes aways once the provider is remove it.

What environment did it happen in?

Crossplane version: v1.5.0
Provider-aws: v0.21.2
Kubernetes: 1.20.11

Any ideas on how we could solve this?

[davidspek]

Related to crossplane/crossplane#2649 and #1076

[haarchri]

in general an Kubernetes issue - please Check references Crossplane issues for enhancements with the beginning of Kubernetes 1.23

[ale-kennedy]

Hi users,
Any idea how to resolve this issue ? I am getting the same error running on EKS 1.21 with Crossplane v1.9.0
Kind regards,

[ivankatliarchuk]

Same. Degradation on EKS API across all the clusters. Not happening when crossplane is removed

[ghost]

It's still happening when using EKS 1.25

[maradwan]

Still facing the issue

[rgarcia6520]

Seeing this in RKE2 1.26 and even in some pods in-cluster that have to make api requests