Wait(n=1) would exceed context deadline

[gfrid]

Describe the bug
My solution consist of 36 containers when i preform Helm upgrade of the solution I get rpc error: code = Unknown desc = us-west-2: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline"

To Reproduce
Deploy many constrainers at once (above 15 at least) at same time in AWS EKS

Steps to reproduce the behavior:
Install AWS EKS 1.28, 1.29
Install the latest CIS drivers and run mega deployment
Run mega deployment with many containers at once - each container has at least 10 mounts from different secret manager secrets

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No
Yes

If yes, the issue is likely with the k8s Secrets Store CSI driver, not the AWS provider. Open an issue in that repo.

Expected behavior
Volumes should mount

Environment:
EKS 1.29

Additional context
Full error : Warning FailedMount 2m8s (x2 over 4m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod production/helm-maste │ │ r-chart-services-576d879f7d-b7692, err: rpc error: code = Unknown desc = us-west-2: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

[jbct]

Hi @gfrid. This is related to kubernetes client throttling that occurs in the Go libraries. See #136 for example. To address parts of this, we've added the ability to configure qps and burst limits in our provider, but you may be throttled by the upper-level driver as well. Since this is occurring with other vendors, you may want to create an enhancement with them. Closing as duplicate of 136.