Since Helm 3.1.0 we got a lot of warnings from awscli that too much clients are created.

[TomasKohout]

Output of helm version:

version.BuildInfo{Version:"v3.1.0", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-12T13:43:46Z", GoVersion:"go1.13.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.9-eks-c0eccc", GitCommit:"c0eccca51d7500bb03b2f163dd8d534ffeb2f7a2", GitTreeState:"clean", BuildDate:"2019-12-22T23:14:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
AWS

When we run install or upgrade then a lot of

W0303 09:43:12.111352    8329 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 1961 clients constructed calling "aws"

occur. We use 1 prehook and 1 posthook. It was not happening with Helm 3.0.x. Does anybody know if this is ok or not? :-)

[jdolitsky]

This appears to be some issue in using client-go to auth against AWS. This is the source of error:
https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/exec/exec.go#L202

I'm wondering if there was some environmental issue with the "aws" plugin (which I'm assuming means the aws-iam-autheticator). I'm wondering if that needs to be updated.

[TomasKohout]

I'll check out if we are outdated. I suppose that my PC is updated, but will check it out tomorrow. :-)

[TomasKohout]

aws-iam-authenticator version

{"Version":"v0.4.0","Commit":"c141eda34ad1b6b4d71056810951801348f8c367"}

It's installed from https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html

[TomasKohout]

@jdolitsky would you know what PR could cause this? Older version of Helm 3 was doing fine and only Helm was updated to 3.1.x

[jdolitsky]

@TomasKohout do you have a previous version of helm (specific 3.x version) that works?

[TomasKohout]

3.0.3 was without those warnings. I've tried all rc versions of 3.1.0. Those warnings occur on all of them.

[TomasKohout]

I think that this is actually a regression so it should be labeled accordingly. Also creating 3000 clients is not a very good practice. Why does Helm needs so much clients?

[TomasKohout]

Looks like for every kubernetes object new client is created. @jdolitsky could you verify this? :)

[bacongobbler]

That should not be the case. The kubernetes client is created once per invocation (during action.Init() and is shared for the entire install operation.

See

helm/pkg/action/action.go

Line 225 in 59447f8

kc := kube.New(getter)

Let us know if you find anything else.

[TomasKohout]

Well, here are the logs from upgrade action.
helm3-0-3.log
helm3-1-2.log

[TomasKohout]

I do not know guys. Maybe new version of kubernetes package? :-)

[bacongobbler]

Thank you for the logs.

I found the source of the log message. That is definitely code which the EKS authenticator would traverse.

Based on the information provided here, I can only see this as an issue coming from the EKS authenticator, which is outside of Helm's control.

I noticed you are running an older version of Kubernetes: 1.14. Have you considered upgrading?

Lastly, I would highly suggest looking at the developer guide to see if you can try to nail the issue down yourself. I have not seen a similar report from another user, and I have not observed this behaviour before. This is why I'm asking to please let us know if you find anything.

I do not know guys. Maybe new version of kubernetes package? :-)

Helm 3.1.2 uses Kubernetes 1.17 core libraries, and HEAD uses 1.18. As far as Helm's concerned, we are keeping up to date with Kubernetes here.

Hope this helps.

[SapientGuardian]

I didn't want to add a "me too" reply, but because you said " I have not seen a similar report from another user"...

I'm on EKS 1.15 with Helm 2.11 upgrading to 3.2.1. I needed to update some of my helm packages to address deprecations in Kubernetes 1.16, and in order to do that I needed to upgrade Helm and figured I'd go all the way to 3.2.1. I'm migrating my packages using https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/ . During the final step of the conversion process (helm3 2to3 cleanup) I got the same warnings, with the final one being W0515 16:20:25.464868 541 exec.go:201] constructing many client instances from the same exec auth config can cause performance problems during cert rotation and can exhaust available network connections; 3034 clients constructed calling "aws". That does seem like a lot of clients :-) .

I haven't seen any adverse effects from this, just wanted to chime in to say that it isn't just that one guy running into it.

[bacongobbler]

Found a similar upstream issue: kubernetes-sigs/aws-iam-authenticator#308.

As mentioned in that ticket, this is out of our control. It's likely a bug within Kubernetes. Please file a ticket upstream.