Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ECR authentication #112

Closed
hobbsh opened this issue Oct 23, 2020 · 57 comments
Closed

Support ECR authentication #112

hobbsh opened this issue Oct 23, 2020 · 57 comments
Labels
enhancement New feature or request
Milestone

Comments

@hobbsh
Copy link

hobbsh commented Oct 23, 2020

Is your feature request related to a problem? Please describe.
It looks like there is a lack of support for getting the API token required to authenticate against the ECR API. It would be great to be able to make use of the IAM Instance Profile assigned to a node which has GetAuthorizationToken permissions. It might look similar to this: https://github.com/awslabs/amazon-ecr-credential-helper/blob/master/ecr-login/api/client.go.

Describe the solution you'd like
The image updater supports ECR authentication

Describe alternatives you've considered
There do not seem to be any alternatives without code changes.

@hobbsh hobbsh added the enhancement New feature or request label Oct 23, 2020
@hobbsh
Copy link
Author

hobbsh commented Oct 23, 2020

For additional context, this is the logged error I get when trying to use ECR:

time="2020-10-23T01:53:29Z" level=error msg="Could not get tags from registry: Get \"https://ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/v2/dev/my-image/tags/list\": http: non-successful response (status=401 body=\"Not Authorized\\n\")" alias=backend application=my-application image_name=dev/my-image image_tag=304dbc93e21898e95272d7cb81b61a671a8b7365 registry=ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com

@hobbsh
Copy link
Author

hobbsh commented Oct 23, 2020

We can use aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' and jam it into a k8s secret (it yields a base64 encoded username:password string) however the token is only valid for 12 hours so there would need to be a cronjob that updates this secret every 12 hours.

@jannfis
Copy link
Contributor

jannfis commented Oct 26, 2020

We currently have three methods for getting credentials right now - secret, pullsecret and env. Would a fourth option, that calls a script and re-uses its output as credentials, be sufficient for this use case?

For example we could introduce a new method ext:/path/to/script.sh, and you could either build your own image from the image updater's docker file or use an init container to copy the script to some location into the image updater's container.

@jannfis jannfis added this to the v0.8.0 milestone Nov 22, 2020
@jannfis
Copy link
Contributor

jannfis commented Nov 23, 2020

A feature implementing my above proposal has been merged with #121 and will be part of v0.8 release.

@jannfis
Copy link
Contributor

jannfis commented Nov 23, 2020

I have no means to test against a real ECR instance, but would a script require some kind of parametrization in order to be able to retrieve credentials for the correct registry?

@vistrcm
Copy link

vistrcm commented Dec 2, 2020

Hello,
I can try to help with testing. ECR support is crucial for me too.

One question before I can test: do credentials cached somehow? I mean, does argocd-image-updater read credentials from the secret, env variable or execute the script every time or only once and then use these results?

It looks like function SetEndpointCredentials sets credentials for RegistryEndpoint and never update that

func (ep *RegistryEndpoint) SetEndpointCredentials(kubeClient *client.KubernetesClient) error {

Sorry, I had no chance to look deeper.

If credentials are cached script will not help with ECR. By default, ECR tokens rotated every 12 hours.

@vistrcm
Copy link

vistrcm commented Dec 3, 2020

It looks like credentials are cached if defined as a parameter of registry configuration and not cached if specified as annotation for the application.

So using that knowledge, I was able to annotate my applications with

annotations:
  argocd-image-updater.argoproj.io/image-list: org/app=XXXXXXXXXXXX.dkr.ecr.region.amazonaws.com/org/app
  argocd-image-updater.argoproj.io/org_app.update-strategy: latest
  argocd-image-updater.argoproj.io/org_app.kustomize.image-name: org/app
  argocd-image-updater.argoproj.io/org_app.pull-secret: secret:argocd-image-updater/aws-ecr-creds#creds

And create simple k8s CronJob to get token and update secret value:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ecr-secret-udpater
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ecr-secret-udpater
rules:
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ecr-secret-udpater
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ecr-secret-udpater
subjects:
  - kind: ServiceAccount
    name: ecr-secret-udpater
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    description: this secret is dynamically updated by the k8s CronJob ecr-secret-update. store ECR registry user/token
  name: aws-ecr-creds
stringData:
  creds: will_be_set_by_the_job
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: ecr-secret-update
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - args:
                - -c
                - kubectl create secret generic aws-ecr-creds --from-literal=creds=AWS:$(cat /store/token) --dry-run=client -o yaml | kubectl replace -f -
              command:
                - sh
              image: org/kubectl:v1.19.4
              name: kubectl
              volumeMounts:
                - mountPath: /store
                  name: store
          initContainers:
            - args:
                - -c
                - aws ecr get-login-password --region us-west-2 > /store/token
              command:
                - sh
              image: amazon/aws-cli:2.1.6
              name: get-login-password
              volumeMounts:
                - mountPath: /store
                  name: store
          restartPolicy: OnFailure
          serviceAccountName: ecr-secret-udpater
          volumes:
            - emptyDir:
                medium: Memory
              name: store
      ttlSecondsAfterFinished: 100
  schedule: '* */6 * * *'

This approach works for me on v0.7.0. I can share AWS related policies I used to grant permissions and details on org/kubectl:v1.19.4 image is someone is interested.

@jannfis
Copy link
Contributor

jannfis commented Dec 3, 2020

@vistrcm Thank you for your input! So I see the problem with the cached credentials.

I think an expiry time for the credentials might help in case of the credentials being cached on registry level, something like a new toggle credentials_expire: <duration> for the registry configuration. If credentials are older than <duration>, they'll be regenerated from its source (i.e. secret read again, or script executed again)

I'm planning to release v0.8 this weekend hopefully, so such a change (if helpful) could make it in there, I believe.

@hobbsh
Copy link
Author

hobbsh commented Dec 3, 2020

I use the same solution currently to update ECR creds and it would be great if the image-updater could handle this itself.

@vistrcm
Copy link

vistrcm commented Dec 3, 2020

I think an expiry time for the credentials might help in case of the credentials being cached on registry level, something like a new toggle credentials_expire: <duration> for the registry configuration. If credentials are older than <duration>, they'll be regenerated from its source (i.e. secret read again, or script executed again)

I'm planning to release v0.8 this weekend hopefully, so such a change (if helpful) could make it in there, I believe.

that would be great! thank you

@jannfis
Copy link
Contributor

jannfis commented Dec 4, 2020

So with #124 merged, you can now specify an expiration time for your credentials, when configured at the registry level (i.e. in registries.conf). So now you can use something like the following:

credentials: ext:/some/where/eks-creds.sh
credexpire: 11h59m

The script at /some/where/eks-creds.sh should be a wrapper to call the aws CLI with all required parametrization, and output the resulting credentials as single line on stdout in the format <username>:<token>. As long as credentials are not expired (or image-updater is restarted), script will not be called and credentials are cached in-memory. After expiration time, the script will be called again to generate a new token.

This is also documented at https://argocd-image-updater.readthedocs.io/en/latest/configuration/registries/

Of course you would have to create an init container that copies required tools into the image updater's container.

Is this something you'll be able to use? I'll not be able to provide native ECR/AWS support, because I don't use AWS.

@hobbsh
Copy link
Author

hobbsh commented Dec 4, 2020

I'll try to test that in the next week or two. There's a typo in the docs, you mention EKS instead of ECR 😉

@jannfis
Copy link
Contributor

jannfis commented Dec 6, 2020

I have released v0.8.0 today. To simplify testing, it introduced a new command test to the argocd-image-updater CLI. This command can also be used to test authentication to registries, without having to run it inside Kubernetes (and therefore possibly build a new image) and without the need to trial&error annotate your Argo CD applications.

You can simply test an authentication script as follows:

argocd-image-updater test <your_image_on_ecr> --credentials ext:/path/to/your/script --registries-conf /path/to/your/registries.conf

It can also do more to help you check image updater's behaviour for your specific images, and is rudimentary documented here

The binary (only linux-amd64 so far) can be downloaded from the release page

@jannfis jannfis modified the milestones: v0.8.0, v0.9.0 Jan 3, 2021
@diranged
Copy link

Just for anyone who finds this and is wondering what the TLDR is ... this seems to work:

registries.conf:

registries:
- name: ECR
  api_url: https://xxx.dkr.ecr.us-west-2.amazonaws.com
  prefix: xxx.dkr.ecr.us-west-2.amazonaws.com
  ping: yes
  insecure: no
  tagsortmode: latest-first
  credentials: ext:/path/to/ecr.sh
  credsexpire: 10h

ecr.sh

#!/bin/sh
aws ecr --region us-west-2 get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d
$ ./dist/argocd-image-updater  test xxx.dkr.ecr.us-west-2.amazonaws.com/dev/xyz-batch-serving --registries-conf ./registries.conf
INFO[0000] getting image                                 image_name=dev/xxx-batch-serving registry=xxx.dkr.ecr.us-west-2.amazonaws.com
DEBU[0000] rate limit for https://xxx.dkr.ecr.us-west-2.amazonaws.com is 2147483647 
INFO[0000] Loaded 1 registry configurations from ./registries.conf 
INFO[0000] git/argocd-image-updater/ecr.sh  dir= execID=c6FS4
INFO[0000] Fetching available tags and metadata from registry  image_name=dev/xyz-batch-serving
INFO[0001] Found 288 tags in registry                    image_name=dev/xxx-batch-serving
DEBU[0001] could not parse input tag abc-xyz as semver: Invalid Semantic Version 
...

@jannfis
Copy link
Contributor

jannfis commented Mar 19, 2021

I will close this issue now, since I think the feature works now. Feel free to reopen it when you think there should be more work done to support ECR auth.

@jannfis jannfis closed this as completed Mar 19, 2021
@jeroenmaas
Copy link

@diranged

Thanks for sharing your config. It worked well, though one small note. I had to change tagsortmode: latest-first to none since our ECR repo doesn't return the tags in order causing latest to not work when the value is specified.

@diranged
Copy link

@diranged

Thanks for sharing your config. It worked well, though one small note. I had to change tagsortmode: latest-first to none since our ECR repo doesn't return the tags in order causing latest to not work when the value is specified.

@jeroenmaas Can you share logs or more of your config showing that tagsortmode: none works for you? I just opened up #216 because it turns out that I am seeing that fail on our side making the API calls to AWS.

@diranged
Copy link

Just following up - the actual issue was a permissions issue. See #216 (comment).

@jvanson
Copy link

jvanson commented Aug 13, 2021

registries:

@diranged How do you install the script ecr.sh? How did you make it so that the argo-cd-image-updater pod can access the script? Thanks!

@AndresJulia
Copy link

Just for anyone who finds this and is wondering what the TLDR is ... this seems to work:

registries.conf:

registries:
- name: ECR
  api_url: https://xxx.dkr.ecr.us-west-2.amazonaws.com
  prefix: xxx.dkr.ecr.us-west-2.amazonaws.com
  ping: yes
  insecure: no
  tagsortmode: latest-first
  credentials: ext:/path/to/ecr.sh
  credsexpire: 10h

ecr.sh

#!/bin/sh
aws ecr --region us-west-2 get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d
$ ./dist/argocd-image-updater  test xxx.dkr.ecr.us-west-2.amazonaws.com/dev/xyz-batch-serving --registries-conf ./registries.conf
INFO[0000] getting image                                 image_name=dev/xxx-batch-serving registry=xxx.dkr.ecr.us-west-2.amazonaws.com
DEBU[0000] rate limit for https://xxx.dkr.ecr.us-west-2.amazonaws.com is 2147483647 
INFO[0000] Loaded 1 registry configurations from ./registries.conf 
INFO[0000] git/argocd-image-updater/ecr.sh  dir= execID=c6FS4
INFO[0000] Fetching available tags and metadata from registry  image_name=dev/xyz-batch-serving
INFO[0001] Found 288 tags in registry                    image_name=dev/xxx-batch-serving
DEBU[0001] could not parse input tag abc-xyz as semver: Invalid Semantic Version 
...

But how do you get awscli inside the container?

@hobbsh
Copy link
Author

hobbsh commented Aug 14, 2021

@diranged How do you install the script ecr.sh? How did you make it so that the argo-cd-image-updater pod can access the script? Thanks!

You will have to mount it from a configmap

@hobbsh
Copy link
Author

hobbsh commented Aug 14, 2021

But how do you get awscli inside the container?

@AndresJulia I have not tried this but I am assuming people using this are extending the argocd-image-updater image to install awscli. Not ideal but that would work.

@sgavrylenko
Copy link

sgavrylenko commented Aug 17, 2021

But how do you get awscli inside the container?

@AndresJulia At this moment, awscli is present in the image

@joebowbeer
Copy link
Contributor

joebowbeer commented Jun 17, 2022

I've been using keel.sh partly because of its excellent ECR integration via IRSA and an AWS-aware credentials helper:

https://github.com/keel-hq/keel/tree/master/extension/credentialshelper/aws

But it would be nice to have another option.

@vikas027
Copy link

I've been using keel.sh partly because of its excellent ECR integration via IRSA and an AWS-aware credentials helper:

Keel hasn't updated for a while, looks like the project has been abandoned :(

@joebowbeer
Copy link
Contributor

Keel hasn't updated for a while, looks like the project has been abandoned :(

I can't disagree but it still functions. keel is my only option at the moment, and if it breaks then someone may need to fork it and fix it. It would be nice to have another option..

@vikas027
Copy link

keel is my only option at the moment

I have not tried Image Updater yet but there are few people who have successfully got the updater working with ECR. Did you try that?

@joebowbeer
Copy link
Contributor

@vikas027 wrote:

there are few people who have successfully got the updater working with ECR

Please interpret my comments as an RFE for ECR integration via IRSA and an AWS-aware credentials helper.

@fabioaraujopt
Copy link

fabioaraujopt commented Aug 1, 2022

I wanted to point out that most of these steps are now simplified by the helm chart. e.g: registries config can be configured as follow:

config:
  registries:
  - name: Docker Hub
    api_url: https://registry-1.docker.io
    prefix: docker.io
  - name: ECR
    api_url: https://xxx.dkr.ecr.us-west-1.amazonaws.com
    prefix: xxx.dkr.ecr.us-west-1.amazonaws.com   # before v0.12 this needed to be set empty, prefix: ""
    default: true  # not supported before v0.12
    ping: yes
    insecure: no
    credentials: ext:/scripts/ecr-login.sh  #script name should match here and in authScripts 
    credsexpire: 11h

and configure ECR authentication script as follow:

authScripts:
  enabled: true
  scripts: 
    ecr-login.sh: |   # notice script name matches above    
      #!/bin/sh
      aws ecr --region $AWS_REGION get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d

In my case, $AWS_REGION is set as environment variable.

extraEnv:
  - name: AWS_REGION
    value: "us-west-1"

This setup will mount the login script as /scripts/ecr-login.sh and will run every 11 hours.

I'm able to run image-updater on EKS with IRSA configured. Again using the current helm version, you will need to enable service account for image-updater and assign a role with appropriate IAM permissions, e.g:

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<Account-ID>:role/<role-name>
  name: "argocd-image-updater"  # I think this is the default

For IAM permissions you will at least need AmazonEC2ContainerRegistryReadOnly.

I hope this helps others get started with image-updater on AWS. @jannfis if you think this is worth adding to the docs, im happy to create a PR.

I'm having this error:

Could not set registry endpoint credentials: error executing /scripts/ecr-login.sh: fork/exec /scripts/ecr-login.sh: no such file or directory"

I'm using the following config:

config:
  registries:
    - name: ECR
      api_url: https://XXX.dkr.ecr.eu-west-1.amazonaws.com
      prefix: XXX.dkr.ecr.eu-west-1.amazonaws.com
      ping: yes
      insecure: no
      credentials: ext:/scripts/ecr-login.sh
      credsexpire: 6h
authScripts:
  enabled: true
  scripts:
    ecr-login.sh: |
      #!/bin/sh
      aws ecr get-authorization-token --region eu-west-1 --registry-ids XXXX --output text --query 'authorizationData[].authorizationToken'

@thuandt
Copy link

thuandt commented Aug 2, 2022

@fabioaraujopt Please ensure ecr-login.sh script exist in argocd-image-updater container (mount as ConfigMap in my case)

@zrice57
Copy link

zrice57 commented Dec 20, 2022

@fabioaraujopt I was having the same issue. I believe the version of the chart which I was installing, 0.8.1, by default uses the same image version and that image version does not contain any sh or bash executables. It is probably the lack of those files resulting in no such file or directory.

I was able to get it working by specifying the latest image version in my values file:

image:
  tag: "v0.12.1"

I'm not sure how other people were getting this to work, perhaps we are using different versions of the helm chart? 0.8.1 appears to be the latest version.

@boris-infinit
Copy link

@mubarak-j
Thank You this is really working example with Helm Chart

@zhaque44
Copy link
Contributor

ConfigMap for ECR authentication

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
  namespace: argocd
data:
  registries.conf: |
    registries:
    - name: AWS ECR
      prefix: <aws_account_id>.dkr.ecr.eu-west-2.amazonaws.com
      api_url: https://<aws_account_id>.dkr.ecr.eu-west-2.amazonaws.com
      credentials: secret:argocd/ecr-secrets#creds
      default: true
      insecure: yes
     credsexpire: 12h

for authenticating to ecr, use the cronjob to generate credentials on every schedule.

do you have an example repo for this?

@mccullya
Copy link

The scripts above i.e.
aws ecr get-authorization-token --region eu-west-1 --registry-ids XXXX --output text --query 'authorizationData[].authorizationToken'

were not working for me. It was erroring saying that it wants a username:password. I used this instead, and it works :)

authScripts:
  enabled: true
  scripts: 
    ecr-login.sh: |
      #!/bin/sh
      # Retrieve the authorization token from AWS ECR
      auth_token=$(aws ecr get-authorization-token --region eu-west-1 --output text --query 'authorizationData[].authorizationToken')
      
      # Decode the authorization token
      decoded_token=$(echo $auth_token | base64 -d)
      
      # Extract username and password
      username=$(echo $decoded_token | cut -d: -f1)
      password=$(echo $decoded_token | cut -d: -f2)
      
      # Output username and password
      echo "$username:$password"

@bradenwright
Copy link

Fwiw I followed this article and it worked for me, https://medium.com/@tomas94depi/argo-image-updater-with-aws-ecr-ddb661abb332

@giogujabidze
Copy link

giogujabidze commented Jul 17, 2024

Hi guys, I'm afraid this workaround/solution stoped working in a recent versions. As currently I'm using the helm chart version 0.11.0 which uses the application v0.14.0 and the script does not work, I guess because of the fact, that the /app directory is mounted as a readonly, to whenever I try to run aws commands from within the pod, it throws the following error message: [Errno 30] Read-only file system: '/app/.aws'

I tried to solve that by using the custom credentials files using aws reserved environment variables:

  extraEnv:
    - name: AWS_REGION
      value: "us-east-1"
    - name: AWS_CONFIG_FILE
      value: "/tmp/.aws/config"
    - name: AWS_SHARED_CREDENTIALS_FILE
      value: "/tmp/.aws/credentials"

After this, if I run 'aws configure' it creates config file under /tmp/.aws directory, but if I run for example 'aws ecr get-authorization-token --region eu-west-1' commnad, it still tries to use the $HOME/.aws directory (which refers to /apps/.aws) and still fails.

Maybe your container image has somehow hardcoded aws configuration file location?

This is my entire configuration (helm value file):

  config:
    applicationsAPIKind: "kubernetes"

    logLevel: "debug"

    registries:
    - name: ECR
      api_url: https://<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
      prefix: <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
      default: true
      ping: yes
      insecure: no
      credentials: ext:/scripts/ecr-login.sh
      credsexpire: 10h

  extraEnv:
    - name: AWS_REGION
      value: "us-east-1"
    - name: AWS_CONFIG_FILE
      value: "/tmp/.aws/config"
    - name: AWS_SHARED_CREDENTIALS_FILE
      value: "/tmp/.aws/credentials"

  serviceAccount:
    create: true
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/argocd-image-updater-role
    name: "argocd-image-updater"

  # whether to mount authentication scripts, if enabled, the authentication scripts will be mounted on /scripts that can be used to authenticate with registries (ECR)
  # refer to https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/#specifying-credentials-for-accessing-container-registries for more info
  authScripts:
    # -- Whether to mount the defined scripts that can be used to authenticate with a registry, the scripts will be mounted at `/scripts`
    enabled: true
    scripts:
      ecr-login.sh: |
        #!/bin/sh
        echo "AWS:$(aws ecr get-login-password --region us-east-1)"

P.S. I tried to use the script described above:
aws ecr --region $AWS_REGION get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d
but faced the issue which already is described here, so found the simpler command (since the username is always the same):
echo "AWS:$(aws ecr get-login-password --region us-east-1)"
But like I said, i cannot test/run any awscli comands from within the argocd-image-updater pod. (((

@mubarak-j
Copy link
Contributor

@giogujabidze it seems the breaking change was introduced in helm chart v0.10.0 which added securityContext with default value readOnlyRootFilesystem: true, you may want to override that in your helm values.

@giogujabidze
Copy link

giogujabidze commented Jul 18, 2024

@giogujabidze it seems the breaking change was introduced in helm chart v0.10.0 which added securityContext with default value readOnlyRootFilesystem: true, you may want to override that in your helm values.

Thanks @mubarak-j for confirming my doubts.
Adding the following section to my helm values file fixed everything:

securityContext:
  readOnlyRootFilesystem: false

@giogujabidze
Copy link

I'm putting the whole helm values file content for others, who will try to use ArgoCD Image Updater against AWS ECR, to not lose as much time as I did to make it work. Here is the configuration that worked for me (helm chart version - 0.11.0):

extraEnv:
  - name: AWS_REGION
    value: "us-east-1"

config:
  applicationsAPIKind: "kubernetes"
  logLevel: "debug"

  registries:
  - name: ECR
    api_url: https://<AWS_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com
    prefix: <AWS_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com
    default: true
    ping: yes
    insecure: no
    credentials: ext:/scripts/ecr-login.sh
    credsexpire: 10h
 
authScripts:
  enabled: true
  scripts:
    ecr-login.sh: |
      #!/bin/sh
      echo "AWS:$(aws ecr get-login-password --region $AWS_REGION)"

serviceAccount:
  create: true
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT>:role/argocd-image-updater-role
  name: "argocd-image-updater"

securityContext:
  readOnlyRootFilesystem: false

@peter-nguyen-rw
Copy link

Hello. I'm still running into this issue regarding : no basic auth credentials

@giogujabidze I have the same exact Helm fields as you posted above. Any ideas?

@giogujabidze
Copy link

Hello. I'm still running into this issue regarding : no basic auth credentials

@giogujabidze I have the same exact Helm fields as you posted above. Any ideas?

Hi. Well it's difficult to say exact reason, as I also spent a lot of time to make it work, but in your case I would check the IRSA policy, maybe you missed some essential permissions and because of that, you cannot actually authenticate against ECR. Here is the list of some possible reasons for this error message.

@peter-nguyen-rw
Copy link

@giogujabidze
Do you mind if I see your IRSA policy and the the policy for the role associated with it? Also can you tell me what Helm targetRevision and image version you're on for the Image Updater please?

@giogujabidze
Copy link

@giogujabidze Do you mind if I see your IRSA policy and the the policy for the role associated with it? Also can you tell me what Helm targetRevision and image version you're on for the Image Updater please?

This is the policy attached to my IRSA role for image updater:

{
    "Statement": [
        {
            "Action": [
                "ecr:ValidatePullThroughCacheRule",
                "ecr:ListTagsForResource",
                "ecr:ListImages",
                "ecr:GetRepositoryPolicy",
                "ecr:GetRegistryScanningConfiguration",
                "ecr:GetLifecyclePolicy",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken",
                "ecr:DescribeRepositories",
                "ecr:DescribeRegistry",
                "ecr:DescribePullThroughCacheRules",
                "ecr:DescribeImages",
                "ecr:DescribeImageScanFindings",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

As for the helm targetRevision, like I already mentioned above, I am using the latest (at this moment) version of helm chart - 0.11.0

@peter-nguyen-rw
Copy link

@giogujabidze
Okay so this is odd. Because the 0.11.0 version of the Helm chart apparently does not support the field default: true within your registries.conf.

See this comment here about that field not being supported for versions before 0.12.0
#112 (comment)

@giogujabidze
Copy link

@giogujabidze Okay so this is odd. Because the 0.11.0 version of the Helm chart apparently does not support the field default: true within your registries.conf.

See this comment here about that field not being supported for versions before 0.12.0 #112 (comment)

I believe that comment was referred to the application version itself, as the helm chart 0.11.0 includes the application version v0.14.0- https://github.com/argoproj/argo-helm/blob/main/charts/argocd-image-updater/Chart.yaml#L6

That means that if you'll deploy this helm chart, it will deploy the argocd image updater docker image version v0.14.0 by default - https://github.com/argoproj/argo-helm/blob/main/charts/argocd-image-updater/templates/deployment.yaml#L130

Anyways, like I said, it works in my case and it's difficult to say, why it's not working for you.

@peter-nguyen-rw
Copy link

@giogujabidze
Okay thanks for that clarification.

Sorry, back to your IAM setup. The policy you pasted to me is directly attached to your IRSA role? You're not using the IRSA role to assume a role that has those policies attached to it?

@giogujabidze
Copy link

Hey, sorry for late reply

The policy you pasted to me is directly attached to your IRSA role?

Yes, the IAM policy is attached to the role, arn of which is set in annotations of service account for image-updater

@Slhttnunlu
Copy link

Slhttnunlu commented Sep 12, 2024

Greetings,

I am trying to auth to ECR repo with IRSA, I created a role using this policy https://docs.aws.amazon.com/AmazonECR/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEC2ContainerRegistryReadOnly
and added trusted policy, and following the examples in the comments I added the role I created to argocd-image-updater serviceaccount as arn, but when I restart my pod I still get auth error,

When I use the auth method with pullsecret, everything goes smoothly, but for 12 hours, can anyone successfully use ECR auth with IAM role?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests