-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support ECR authentication #112
Comments
For additional context, this is the logged error I get when trying to use ECR:
|
We can use |
We currently have three methods for getting credentials right now - For example we could introduce a new method |
A feature implementing my above proposal has been merged with #121 and will be part of v0.8 release. |
I have no means to test against a real ECR instance, but would a script require some kind of parametrization in order to be able to retrieve credentials for the correct registry? |
Hello, One question before I can test: do credentials cached somehow? I mean, does argocd-image-updater read credentials from the secret, env variable or execute the script every time or only once and then use these results? It looks like function argocd-image-updater/pkg/registry/registry.go Line 168 in 6723a25
Sorry, I had no chance to look deeper. If credentials are cached script will not help with ECR. By default, ECR tokens rotated every 12 hours. |
It looks like credentials are cached if defined as a parameter of registry configuration and not cached if specified as annotation for the application. So using that knowledge, I was able to annotate my applications with annotations:
argocd-image-updater.argoproj.io/image-list: org/app=XXXXXXXXXXXX.dkr.ecr.region.amazonaws.com/org/app
argocd-image-updater.argoproj.io/org_app.update-strategy: latest
argocd-image-updater.argoproj.io/org_app.kustomize.image-name: org/app
argocd-image-updater.argoproj.io/org_app.pull-secret: secret:argocd-image-updater/aws-ecr-creds#creds And create simple k8s CronJob to get token and update secret value: apiVersion: v1
kind: ServiceAccount
metadata:
name: ecr-secret-udpater
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: ecr-secret-udpater
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ecr-secret-udpater
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ecr-secret-udpater
subjects:
- kind: ServiceAccount
name: ecr-secret-udpater
---
apiVersion: v1
kind: Secret
metadata:
annotations:
description: this secret is dynamically updated by the k8s CronJob ecr-secret-update. store ECR registry user/token
name: aws-ecr-creds
stringData:
creds: will_be_set_by_the_job
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: ecr-secret-update
spec:
jobTemplate:
spec:
template:
spec:
containers:
- args:
- -c
- kubectl create secret generic aws-ecr-creds --from-literal=creds=AWS:$(cat /store/token) --dry-run=client -o yaml | kubectl replace -f -
command:
- sh
image: org/kubectl:v1.19.4
name: kubectl
volumeMounts:
- mountPath: /store
name: store
initContainers:
- args:
- -c
- aws ecr get-login-password --region us-west-2 > /store/token
command:
- sh
image: amazon/aws-cli:2.1.6
name: get-login-password
volumeMounts:
- mountPath: /store
name: store
restartPolicy: OnFailure
serviceAccountName: ecr-secret-udpater
volumes:
- emptyDir:
medium: Memory
name: store
ttlSecondsAfterFinished: 100
schedule: '* */6 * * *' This approach works for me on v0.7.0. I can share AWS related policies I used to grant permissions and details on org/kubectl:v1.19.4 image is someone is interested. |
@vistrcm Thank you for your input! So I see the problem with the cached credentials. I think an expiry time for the credentials might help in case of the credentials being cached on registry level, something like a new toggle I'm planning to release v0.8 this weekend hopefully, so such a change (if helpful) could make it in there, I believe. |
I use the same solution currently to update ECR creds and it would be great if the image-updater could handle this itself. |
that would be great! thank you |
So with #124 merged, you can now specify an expiration time for your credentials, when configured at the registry level (i.e. in credentials: ext:/some/where/eks-creds.sh
credexpire: 11h59m The script at This is also documented at https://argocd-image-updater.readthedocs.io/en/latest/configuration/registries/ Of course you would have to create an init container that copies required tools into the image updater's container. Is this something you'll be able to use? I'll not be able to provide native ECR/AWS support, because I don't use AWS. |
I'll try to test that in the next week or two. There's a typo in the docs, you mention |
I have released v0.8.0 today. To simplify testing, it introduced a new command You can simply test an authentication script as follows: argocd-image-updater test <your_image_on_ecr> --credentials ext:/path/to/your/script --registries-conf /path/to/your/registries.conf It can also do more to help you check image updater's behaviour for your specific images, and is rudimentary documented here The binary (only linux-amd64 so far) can be downloaded from the release page |
Just for anyone who finds this and is wondering what the TLDR is ... this seems to work: registries.conf:
ecr.sh
|
I will close this issue now, since I think the feature works now. Feel free to reopen it when you think there should be more work done to support ECR auth. |
Thanks for sharing your config. It worked well, though one small note. I had to change tagsortmode: latest-first to none since our ECR repo doesn't return the tags in order causing latest to not work when the value is specified. |
@jeroenmaas Can you share logs or more of your config showing that |
Just following up - the actual issue was a permissions issue. See #216 (comment). |
@diranged How do you install the script ecr.sh? How did you make it so that the argo-cd-image-updater pod can access the script? Thanks! |
But how do you get awscli inside the container? |
You will have to mount it from a configmap |
@AndresJulia I have not tried this but I am assuming people using this are extending the |
@AndresJulia At this moment, awscli is present in the image |
I've been using keel.sh partly because of its excellent ECR integration via IRSA and an AWS-aware credentials helper: https://github.com/keel-hq/keel/tree/master/extension/credentialshelper/aws But it would be nice to have another option. |
Keel hasn't updated for a while, looks like the project has been abandoned :( |
I can't disagree but it still functions. keel is my only option at the moment, and if it breaks then someone may need to fork it and fix it. It would be nice to have another option.. |
I have not tried Image Updater yet but there are few people who have successfully got the updater working with ECR. Did you try that? |
@vikas027 wrote:
Please interpret my comments as an RFE for ECR integration via IRSA and an AWS-aware credentials helper. |
I'm having this error:
I'm using the following config:
|
@fabioaraujopt Please ensure |
@fabioaraujopt I was having the same issue. I believe the version of the chart which I was installing, I was able to get it working by specifying the latest image version in my
I'm not sure how other people were getting this to work, perhaps we are using different versions of the helm chart? |
@mubarak-j |
do you have an example repo for this? |
The scripts above i.e. were not working for me. It was erroring saying that it wants a username:password. I used this instead, and it works :)
|
Fwiw I followed this article and it worked for me, https://medium.com/@tomas94depi/argo-image-updater-with-aws-ecr-ddb661abb332 |
Hi guys, I'm afraid this workaround/solution stoped working in a recent versions. As currently I'm using the helm chart version I tried to solve that by using the custom credentials files using aws reserved environment variables: extraEnv:
- name: AWS_REGION
value: "us-east-1"
- name: AWS_CONFIG_FILE
value: "/tmp/.aws/config"
- name: AWS_SHARED_CREDENTIALS_FILE
value: "/tmp/.aws/credentials" After this, if I run 'aws configure' it creates config file under Maybe your container image has somehow hardcoded aws configuration file location? This is my entire configuration (helm value file): config:
applicationsAPIKind: "kubernetes"
logLevel: "debug"
registries:
- name: ECR
api_url: https://<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
prefix: <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com
default: true
ping: yes
insecure: no
credentials: ext:/scripts/ecr-login.sh
credsexpire: 10h
extraEnv:
- name: AWS_REGION
value: "us-east-1"
- name: AWS_CONFIG_FILE
value: "/tmp/.aws/config"
- name: AWS_SHARED_CREDENTIALS_FILE
value: "/tmp/.aws/credentials"
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/argocd-image-updater-role
name: "argocd-image-updater"
# whether to mount authentication scripts, if enabled, the authentication scripts will be mounted on /scripts that can be used to authenticate with registries (ECR)
# refer to https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/#specifying-credentials-for-accessing-container-registries for more info
authScripts:
# -- Whether to mount the defined scripts that can be used to authenticate with a registry, the scripts will be mounted at `/scripts`
enabled: true
scripts:
ecr-login.sh: |
#!/bin/sh
echo "AWS:$(aws ecr get-login-password --region us-east-1)" P.S. I tried to use the script described above: |
@giogujabidze it seems the breaking change was introduced in helm chart v0.10.0 which added |
Thanks @mubarak-j for confirming my doubts. securityContext:
readOnlyRootFilesystem: false |
I'm putting the whole helm values file content for others, who will try to use ArgoCD Image Updater against AWS ECR, to not lose as much time as I did to make it work. Here is the configuration that worked for me (helm chart version - extraEnv:
- name: AWS_REGION
value: "us-east-1"
config:
applicationsAPIKind: "kubernetes"
logLevel: "debug"
registries:
- name: ECR
api_url: https://<AWS_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com
prefix: <AWS_ACCOUNT>.dkr.ecr.us-east-1.amazonaws.com
default: true
ping: yes
insecure: no
credentials: ext:/scripts/ecr-login.sh
credsexpire: 10h
authScripts:
enabled: true
scripts:
ecr-login.sh: |
#!/bin/sh
echo "AWS:$(aws ecr get-login-password --region $AWS_REGION)"
serviceAccount:
create: true
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT>:role/argocd-image-updater-role
name: "argocd-image-updater"
securityContext:
readOnlyRootFilesystem: false |
Hello. I'm still running into this issue regarding : no basic auth credentials @giogujabidze I have the same exact Helm fields as you posted above. Any ideas? |
Hi. Well it's difficult to say exact reason, as I also spent a lot of time to make it work, but in your case I would check the IRSA policy, maybe you missed some essential permissions and because of that, you cannot actually authenticate against ECR. Here is the list of some possible reasons for this error message. |
@giogujabidze |
This is the policy attached to my IRSA role for image updater: {
"Statement": [
{
"Action": [
"ecr:ValidatePullThroughCacheRule",
"ecr:ListTagsForResource",
"ecr:ListImages",
"ecr:GetRepositoryPolicy",
"ecr:GetRegistryScanningConfiguration",
"ecr:GetLifecyclePolicy",
"ecr:GetDownloadUrlForLayer",
"ecr:GetAuthorizationToken",
"ecr:DescribeRepositories",
"ecr:DescribeRegistry",
"ecr:DescribePullThroughCacheRules",
"ecr:DescribeImages",
"ecr:DescribeImageScanFindings",
"ecr:BatchGetRepositoryScanningConfiguration",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
}
],
"Version": "2012-10-17"
} As for the helm targetRevision, like I already mentioned above, I am using the latest (at this moment) version of helm chart - |
@giogujabidze See this comment here about that field not being supported for versions before 0.12.0 |
I believe that comment was referred to the application version itself, as the helm chart That means that if you'll deploy this helm chart, it will deploy the argocd image updater docker image version v0.14.0 by default - https://github.com/argoproj/argo-helm/blob/main/charts/argocd-image-updater/templates/deployment.yaml#L130 Anyways, like I said, it works in my case and it's difficult to say, why it's not working for you. |
@giogujabidze Sorry, back to your IAM setup. The policy you pasted to me is directly attached to your IRSA role? You're not using the IRSA role to assume a role that has those policies attached to it? |
Hey, sorry for late reply
Yes, the IAM policy is attached to the role, arn of which is set in annotations of service account for image-updater |
Greetings, I am trying to auth to ECR repo with IRSA, I created a role using this policy https://docs.aws.amazon.com/AmazonECR/latest/userguide/security-iam-awsmanpol.html#security-iam-awsmanpol-AmazonEC2ContainerRegistryReadOnly When I use the auth method with pullsecret, everything goes smoothly, but for 12 hours, can anyone successfully use ECR auth with IAM role? |
Is your feature request related to a problem? Please describe.
It looks like there is a lack of support for getting the API token required to authenticate against the ECR API. It would be great to be able to make use of the IAM Instance Profile assigned to a node which has
GetAuthorizationToken
permissions. It might look similar to this: https://github.com/awslabs/amazon-ecr-credential-helper/blob/master/ecr-login/api/client.go.Describe the solution you'd like
The image updater supports ECR authentication
Describe alternatives you've considered
There do not seem to be any alternatives without code changes.
The text was updated successfully, but these errors were encountered: