Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm dependency update extremely slow when authenticated with EC2 role #132

Open
angeloudy opened this issue Jan 13, 2021 · 6 comments
Open

Comments

@angeloudy
Copy link

angeloudy commented Jan 13, 2021

I use gitlab-runner to build lots of my helm charts. Lots of the pipeline would run helm dependency update.
I recently rebuilt the docker image that does the job. The helm s3 plugin was upgraded to the latest because I didn't specify the version number.

Now the helm dependency update would take 16 minutes to finish, which used to take less than two minutes.
After some investigation I found that the slowiness was caused by authenticatint through EC2 role.
If I add ~/.aws/credentials to the docker image, helm dependency update finishes instantly.

@ozbillwang
Copy link

ozbillwang commented Jan 13, 2021

Something relative with this PR?

#24

aws sdk go latest version is v1.36.25 already:

https://github.com/aws/aws-sdk-go/releases/tag/v1.36.25

ours is

	github.com/aws/aws-sdk-go v1.27.0

helm-s3/go.mod

Line 11 in 5312896

github.com/aws/aws-sdk-go v1.27.0

@ozbillwang
Copy link

could you build the plugin by yourself with latest aws sdk go version v1.36.25 and try helm dependency update again on gitlab runner?

@diversario
Copy link

I'm not seeing such extreme slowness, but working with S3 repository is noticeably slower - it went from few seconds to ~1 min.

@gokuatkai
Copy link

gokuatkai commented Mar 13, 2021

We are noticing an extremely significant slow down on version 0.9.2 and 10.0.0. up to 12times.
Our deploy ci-jobs using 0.9.1 and earlier run for about 30s, while 0.9.2+ up to 7min.
Job are run on k8s and use the node EC2 role for credentials

could it maybe be due to the new s3 downloader thingy: https://github.com/aws/aws-sdk-go/pull/2823/files ?

@gokuatkai
Copy link

gokuatkai commented Mar 13, 2021

I can confirm, that when using the AWS keys of an IAM user directly, the apply is again fast. I guess we need to look closer in this aws sdk lib to find out what is hapening.
and I found this: aws/aws-sdk-go#2972

@LuisPiedra
Copy link

In our case, a simple helm repo add goes from 400ms in version 0.9.1 to 20s in 0.9.2 and 0.10.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants