Does not work with aws service accounts #280

evilezh · 2019-10-19T01:39:10Z

I tried to run with AWS service account, but couldn't get it work.

I did check on dependencies and as per this page: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html it seems 1.23.13 is minimum version required for it to work.

Probably just simple aws sdk dependency update would work.

andyspiers · 2019-11-05T16:16:20Z

It seems the AWS storage provider is implemented in a separate package chartmuseum/storage

The master branch was upgraded to a suitable AWS SDK version in chartmuseum/storage#27 but this hasn't yet been released.

I've asked for it to be released in chartmuseum/storage#32

Once that is done, it will be possible to upgrade chartmuseum.

jdolitsky · 2020-02-24T20:42:35Z

@evilezh @andyspiers @NikolayMarusenko @biradrags @Igor-Kalyniak @gopisaba @mvisonneau @uLan08 @italolelis Would one of you be willing to test out that this is now fixed on master?

You can use Docker image chartmuseum/chartmuseum:latest@sha256:75dd03cc7111cd5bc6fac87429a09c9f91e30d0acaeaf2cb2d7ac3568344d83d

Once I can get confirmation that this is finally fixed (as well as #152), I'll go ahead and release chartmuseum v1.12.0. Thanks in advance!

italolelis · 2020-02-27T10:53:20Z

Hi @jdolitsky,

I tested the new image with a helm installation:

image:
  repository: chartmuseum/chartmuseum
  tag: latest@sha256:75dd03cc7111cd5bc6fac87429a09c9f91e30d0acaeaf2cb2d7ac3568344d83d
  pullPolicy: IfNotPresent

Unfortunately, I still got the same error. This is the output:

{"L":"ERROR","T":"2020-02-27T10:40:21.818Z","M":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","repo":""}
2020-02-27 10:40:21.818479 I | NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The chartmuseum service account looks like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxx:role/chartmuseum-role
  labels:
    app: chartmuseum
    chart: chartmuseum-2.6.1
    heritage: Helm
    release: chartmuseum
  name: chartmuseum-chartmuseum
  namespace: chartmuseum

I made sure that this cluster supports IRSA properly, and it does as other tools are able to fetch IAMs from service accounts. Maybe there is still something missing to configure?

Happy to help with whatever you need.

Thanks,

jdolitsky · 2020-02-27T16:24:31Z

@italolelis appreciate you trying this out. Looks like maybe updating the module isnt enough, need to perhaps use a different method for auth in the Go code? If anyone has experience with this lmk

bqnguyen94 · 2020-02-28T03:21:46Z

session.New seems to be the issue here, please take a look at aws/aws-sdk-go#2828 and this comment. So essentially we just need to either switch to session.NewSession or simply add AWS_SDK_LOAD_CONFIG=1 for current deployment. Can someone verify if session.NewSession work or I'm happy to try it out once I have time?

sc250024 · 2020-03-05T12:47:21Z

I just tried installing ChartMuseum v0.11.0 with Helm chart stable/chartmuseum v2.8.0. I tried adding:

env:
  open:
    AWS_SDK_LOAD_CONFIG: "1"

But this didn't work either. Still getting the following messages in the logs:

{"L":"ERROR","T":"2020-03-05T12:47:00.455Z","M":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","repo":""}
2020-03-05 12:47:00.455499 I | NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

jdolitsky · 2020-03-09T22:13:27Z

@sc250024 im wondering what happens with this combo and latest image? chartmuseum/chartmuseum:latest@sha256:75dd03cc7111cd5bc6fac87429a09c9f91e30d0acaeaf2cb2d7ac3568344d83d

sc250024 · 2020-03-10T10:43:38Z

@jdolitsky Ah right. I just tried it with the image SHA, and it also didn't work. Same error as above.

EDIT: Totally forgot the AWS_SDK_LOAD_CONFIG: "1" part when I first wrote this. It works with the image SHA if you include that environment variable too 😊

Roughly how long until we can get an official release?

italolelis · 2020-03-20T10:42:54Z

I also can confirm that running the image chartmuseum/chartmuseum:latest@sha256:75dd03cc7111cd5bc6fac87429a09c9f91e30d0acaeaf2cb2d7ac3568344d83d and adding AWS_SDK_LOAD_CONFIG: "1" does the trick. Hopefully we can get a tagged image soon.

Thanks for working on it!

jdolitsky · 2020-03-27T02:13:00Z

Hi everyone, thanks for your patience. New version of ChartMuseum has been released (v0.12.0) with the latest version of AWS SDK. Please try it out.

Closing this issue for now, please comment here if you continue to see any issues.

sc250024 · 2020-03-27T07:29:46Z

Hi everyone, thanks for your patience. New version of ChartMuseum has been released (v0.12.0) with the latest version of AWS SDK. Please try it out.

Closing this issue for now, please comment here if you continue to see any issues.

Thank you!

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com>

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com> Signed-off-by: Ye Liu <ye.liu@hp.com>

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com> Signed-off-by: molinber <bertrand.molin@credit-agricole.net>

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com>

ridicule777 · 2020-10-05T15:06:10Z

Hi @jdolitsky ,

I have the exact same problem. These are my helm values.

env:
  open:
    AWS_SDK_LOAD_CONFIG: true
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: bucket-name
    STORAGE_AMAZON_PREFIX:
    STORAGE_AMAZON_REGION: region-name
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::${account-id}:role/${role-name}"

Tried with AWS_SDK_LOAD_CONFIG: true and AWS_SDK_LOAD_CONFIG: 1. No luck.

jdolitsky · 2020-10-05T15:07:47Z

@ridicule777 - can you confirm you're using chartmuseum 0.12 ?

ridicule777 · 2020-10-05T15:17:17Z

@jdolitsky

Yes Iam using 0.12. Please check the complete helm values.

replicaCount: 3
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
image:
  repository: chartmuseum/chartmuseum
  tag: v0.12.0
  pullPolicy: IfNotPresent
secret:
  labels:
    name: chartmuseum
env:
  open:
    AWS_SDK_LOAD_CONFIG: "1"
    STORAGE: amazon
    STORAGE_AMAZON_BUCKET: ${bucket-name}
    STORAGE_AMAZON_PREFIX:
    STORAGE_AMAZON_REGION: ${region-name}
    CHART_POST_FORM_FIELD_NAME: chart
    PROV_POST_FORM_FIELD_NAME: prov
    DEPTH: 0
    DEBUG: true
    LOG_JSON: true
    DISABLE_STATEFILES: false
    DISABLE_METRICS: false
    DISABLE_API: false
    ALLOW_OVERWRITE: false
    AUTH_ANONYMOUS_GET: false
    INDEX_LIMIT: 0
  secret:
    BASIC_AUTH_USER: somerandomuser
    BASIC_AUTH_PASS: somerandompassword
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::${account-id}:role/${role-name}"

ridicule777 · 2020-10-05T15:22:41Z

Here is the pod log.

kubectl logs -f chartmuseum-chartmuseum-68cdcdd7cf-g4vdb                     
{"L":"DEBUG","T":"2020-10-05T15:13:39.875Z","M":"Fetching chart list from storage","repo":""}
{"L":"ERROR","T":"2020-10-05T15:13:40.067Z","M":"WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: 06a0db2e-5d08-483d-bccc-980493a26209","repo":""}
2020-10-05 15:13:40.067214 I | WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
	status code: 403, request id: 06a0db2e-5d08-483d-bccc-980493a26209

I attached the same service account to a nginx deployment. Installed python, python-pip and awscli.
Did these below mentioned steps.

$ JQ=/usr/bin/jq && curl https://stedolan.github.io/jq/download/linux64/jq > $JQ && chmod +x $JQ
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py
$ pip install awscli --upgrade

$ aws sts assume-role-with-web-identity \
 --role-arn $AWS_ROLE_ARN \
 --role-session-name mh9test \
 --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
 --duration-seconds 1000 > /tmp/irp-cred.txt
$ export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")"
$ export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")"
$ export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")"

And then tried copying a blank file into the s3 bucket, I was able to upload it. So, I'm pretty sure, service account is working.

mbelang · 2020-10-28T16:43:27Z

I managed to make it work base on comments #328 (comment)

env:
  open:
    AWS_SDK_LOAD_CONFIG: true

serviceAccount: 
  create: true
  annotations: 
    "eks.amazonaws.com/role-arn": myRoleArn

securityContext: 
  enabled: true
  fsGroup: 65534

jdolitsky mentioned this issue Feb 24, 2020

index.yaml regenerating every request even without any package updated #152

Closed

jdolitsky closed this as completed Mar 27, 2020

msiebuhr mentioned this issue Jun 15, 2020

Chartmuseum: Set AWS_SDK_LOAD_CONFIG to make S3 work in more cases goharbor/harbor-helm#624

Merged

ninjadq added a commit to ninjadq/harbor that referenced this issue Jun 30, 2020

Fix Amazon S3 storage not work

d0ddd61

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com>

ninjadq mentioned this issue Jun 30, 2020

Fix Amazon S3 storage not work goharbor/harbor#12357

Merged

cafeliker pushed a commit to cafeliker/harbor that referenced this issue Jul 14, 2020

Fix Amazon S3 storage not work

6f6e505

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com> Signed-off-by: Ye Liu <ye.liu@hp.com>

bmfp pushed a commit to bmfp/harbor that referenced this issue Jul 22, 2020

Fix Amazon S3 storage not work

954b279

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com> Signed-off-by: molinber <bertrand.molin@credit-agricole.net>

tedgxt pushed a commit to tedgxt/harbor that referenced this issue Aug 11, 2020

Fix Amazon S3 storage not work

c871352

The Chartmuseum S3 client need set an Env variable Ref: helm/chartmuseum#280 Signed-off-by: DQ <dengq@vmware.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Does not work with aws service accounts #280

Does not work with aws service accounts #280

evilezh commented Oct 19, 2019

andyspiers commented Nov 5, 2019

jdolitsky commented Feb 24, 2020

italolelis commented Feb 27, 2020

jdolitsky commented Feb 27, 2020

bqnguyen94 commented Feb 28, 2020

sc250024 commented Mar 5, 2020 •

edited

Loading

jdolitsky commented Mar 9, 2020

sc250024 commented Mar 10, 2020 •

edited

Loading

italolelis commented Mar 20, 2020

jdolitsky commented Mar 27, 2020

sc250024 commented Mar 27, 2020

ridicule777 commented Oct 5, 2020 •

edited

Loading

jdolitsky commented Oct 5, 2020

ridicule777 commented Oct 5, 2020

ridicule777 commented Oct 5, 2020 •

edited

Loading

mbelang commented Oct 28, 2020

Does not work with aws service accounts #280

Does not work with aws service accounts #280

Comments

evilezh commented Oct 19, 2019

andyspiers commented Nov 5, 2019

jdolitsky commented Feb 24, 2020

italolelis commented Feb 27, 2020

jdolitsky commented Feb 27, 2020

bqnguyen94 commented Feb 28, 2020

sc250024 commented Mar 5, 2020 • edited Loading

jdolitsky commented Mar 9, 2020

sc250024 commented Mar 10, 2020 • edited Loading

italolelis commented Mar 20, 2020

jdolitsky commented Mar 27, 2020

sc250024 commented Mar 27, 2020

ridicule777 commented Oct 5, 2020 • edited Loading

jdolitsky commented Oct 5, 2020

ridicule777 commented Oct 5, 2020

ridicule777 commented Oct 5, 2020 • edited Loading

mbelang commented Oct 28, 2020

sc250024 commented Mar 5, 2020 •

edited

Loading

sc250024 commented Mar 10, 2020 •

edited

Loading

ridicule777 commented Oct 5, 2020 •

edited

Loading

ridicule777 commented Oct 5, 2020 •

edited

Loading