Could not get tags from Google Artifact Registry

[zuccon]

Describe the bug
argocd-image-updater is unable to get the tag list from Google Artifact Registry.

Behaviour can be reproduced running the binary locally or on Kubernetes.

Getting the image tags list using the podman CLI ( podman image search --list-tags ) returns the expected results.

To Reproduce
Steps to reproduce the behavior:

$ gcloud auth print-access-token | podman login -u oauth2accesstoken --password-stdin europe-docker.pkg.dev

$ podman image search --list-tags  europe-docker.pkg.dev/project-id/repo-name/image-name

$ argocd-image-updater test  europe-docker.pkg.dev/project-id/repo-name/image-name
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_name=europe-docker.pkg.dev/project-id/repo-name/image-name registry_url=europe-docker.pkg.dev
DEBU[0000] setting rate limit to 20 requests per second  prefix=europe-docker.pkg.dev registry="https://europe-docker.pkg.dev"
DEBU[0000] Inferred registry from prefix europe-docker.pkg.dev to use API https://europe-docker.pkg.dev
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_name=europe-docker.pkg.dev/project-id/repo-name/image-name registry_url=europe-docker.pkg.dev
FATA[0000] could not get tags: Get "https://europe-docker.pkg.dev/v2/project-id/repo-name/image-name/tags/list": denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/project-id/locations/europe/repositories/repo-name" (or it may not exist)  application=test image_alias= image_name=europe-docker.pkg.dev/project-id/repo-name/image-name registry_url=europe-docker.pkg.dev

Expected behavior
The tag list is successfully fetched.

Version
local: v0.12.2
k8s: quay.io/argoprojlabs/argocd-image-updater:v0.12.0

Logs
From the Kubernetes pod:

"Could not get tags from registry: Get \"https://europe-docker.pkg.dev/v2/project-id/repo-name/image-name/tags/list\": denied: Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"projects/project-id/locations/europe/repositories/repo-name\" (or it may not exist)" alias= application=app-name image_name=project-id/repo-name/image-name image_tag=latest registry=europe-docker.pkg.dev"

[pjhampton]

We are also running into this issue after migrating from GCR to Artifact Registry.

@jannfis This is going to need to be patched anyway for the merge to ArgoCD, and with GCR being sunset in favour of Artifact Registry. If I was to take the time to submit a PR patching this, would it get merged?

[jaykay]

Same here with WorkloadIdentity enabled from within GKE cluster:

Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource

Another pod with gcloud in it works as expected using the same ServiceAccount.

[ngodec]

Ran into the same issue with the latest version of Argo (following this bug). Looking in the logs, the identity is not passed to the Docker-GetTags request; see empty authenticationInfo:

authenticationInfo: {
}
authorizationInfo: [
0: {
  permission: "artifactregistry.repositories.downloadArtifacts"
  resource: "projects/project-id/locations/europe-west2/repositories/my-repo"
}]
requestMetadata: {
  callerIp: "gce-internal-ip" 
  callerSuppliedUserAgent: "Go-http-client/1.1,gzip(gfe)"
}

Also a bit odd to see v2 in the API here, I only see reference to v1 in the docs:

request: {
  @type: "type.googleapis.com/google.logging.type.HttpRequest" 
  requestMethod: "GET"
  requestUrl: "/v2/project-id/my-repo/example-chart-dev/tags/list"
}

[ralphotowo]

We've also encountered this issue after switching to Artifacts Registry. Did anyone find a solution to this yet?

Update: My solution was to update the configMap as follows:

- name: Google Artifacts Registry - us-east4
  api_url: https://us-east4-docker.pkg.dev
  prefix: us-east4-docker.pkg.dev
  ping: no
  credentials: ext:/app/config/script.sh
  credsexpire: 15m

In my case, we use a script for authentication using workload identity, but it should also work if you create a service account key file, package it into a secret and reference it as shown in the documentation.

  credentials: secret:foo/bar#creds

[jaykay]

Thx @ralphotowo for pointing me to the right answer. Works for me now.

For everyone else using WorkloadIdentity one hint which cost me some hours of investigation and head-on-the-table-moments:

A script getting an token from the metadata api eg. http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token is returned an access token. What noone (only the GCP docs somewhere hidden ;-) ) tell you is, that the username is oauth2accesstoken

So to be clear:
If using WorkloadIdentity...
... add a script which
... gets the token from the metadata endpoint and
... returns something like echo "oauth2accesstoken:<yourtoken>"

Hope this helps.

[stefangluszek]

Thx @ralphotowo for pointing me to the right answer. Works for me now.

For everyone else using WorkloadIdentity one hint which cost me some hours of investigation and head-on-the-table-moments:

A script getting an token from the metadata api eg. http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token is returned an access token. What noone (only the GCP docs somewhere hidden ;-) ) tell you is, that the username is oauth2accesstoken

So to be clear: If using WorkloadIdentity... ... add a script which ... gets the token from the metadata endpoint and ... returns something like echo "oauth2accesstoken:<yourtoken>"

Hope this helps.

Thanks @jaykay !
This solution seems to work, at least with argocd-image-updater

GCLOUD_CREDS="oauth2accesstoken:$(gcloud auth print-access-token)" argocd-image-updater test europe-docker.pkg.dev/my-repository/my-helm/my-x --credentials env:GCLOUD_CREDS
DEBU[0000] Creating in-cluster Kubernetes client        
INFO[0000] retrieving information about image            image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
DEBU[0000] setting rate limit to 20 requests per second  prefix=europe-docker.pkg.dev registry="https://europe-docker.pkg.dev"
DEBU[0000] Inferred registry from prefix europe-docker.pkg.dev to use API https://europe-docker.pkg.dev 
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
INFO[0000] Found 21 tags in registry                     application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
DEBU[0000] found 21 from 21 tags eligible for consideration  image=europe-docker.pkg.dev/my-repository/my-helm/my-x
INFO[0000] latest image according to constraint is europe-docker.pkg.dev/my-repository/my-helm/kubernator-x:0.0.21  application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev

Now I need to figure out how to incorporate this into my ArgoCD configuration.

[stefangluszek]

I have the same issue but with helm charts instead of the docker images, so I guess I can not use the argocd-image-updater as mentioned by @jaykay

[stefangluszek]

I have the same issue but with helm charts instead of the docker images, so I guess I can not use the argocd-image-updater as mentioned by @jaykay

I finally seem to have figured it out by debugging the oras-go client. It turns out the provided credentials in the Repository setting where not used because oras-go compares the target with the registry to see if it can use they match. If they do not match EmptyCredential is removed.
In my case I configured the repository URL as: europe-docker.pkg.dev/my-repository/my-helm but the target evaluated to: europe-docker.pkg.dev.
I had to add the registry without the path: europe-docker.pkg.dev and in application sources put the full path in the chart e.g:

- chart: my-repository/my-helm/my-x
      repoURL: europe-docker.pkg.dev

Which just feels wrong. Should I create another issue for this? I am guessing it's the same exact case for the argocd-image-updater? I have not checked though.

[legal90]

@stefangluszek Thank you for the details!

Now I need to figure out how to incorporate this into my ArgoCD configuration.

Did you manage to configure ArgoCD Image Updater to make it working with container images (not Helm charts) stored in GAR? Could you please share your solution?

[legal90]

Nevermind, I found the solution here: #319 (comment) (thanks to @bkanuka! 🎉 )

In case if ArgoCD Image Updater is installed using a Helm chart, the setup for GAR auth via Workload Identity will look like this (example of your custom helm values for argocd-image-updater chart):

# values.yaml 

config:
  registries:
    - name: Google Artifacts Registry - eu-west1
      api_url: https://europe-docker.pkg.dev
      prefix: europe-docker.pkg.dev
      credentials: ext:/scripts/gke-workload-identity-auth.sh
      credsexpire: 30m

authScripts:
  enabled: true
  scripts:
    gke-workload-identity-auth.sh: |
      #!/bin/sh
      ACCESS_TOKEN=$(wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -q -O - | grep -Eo '"access_token":.*?[^\\]",' | cut -d '"' -f 4)
      echo "oauth2accesstoken:$ACCESS_TOKEN"

[smilelikeshit]

thank you very much @legal90 you save my time.

[dtomlinson91]

@legal90 You saved me at least a day's work messing around with this, thank you!

[popchop3]

@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue:

denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist)

When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to.

I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one.

Do I need to mount a volume as mentioned here by @bkanuk: #319 (comment) ?

[popchop3]

@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue:

denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist)

When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to.

I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one.

Do I need to mount a volume as mentioned here by @bkanuk: #319 (comment) ?

Figured this out - I had to delete the whole deployment and not just the pod for this to show in the containers file system.
However, still getting a permission error when trying to access my Artifact Registry repository. Interestingly, when I run a "gcloud auth print-access-token" command and save the token details, it looks like this (much shorter token and this works when I try to access my registry)
ya29.a0AfB_byA_q-IsTFhKXdjs32Q3vMVsQ6v1jw1u0BHr7g9Rz_YrEUt4BSJ6Ssvo3jHKd83ld0Utqrk9zLBSahL2G9KBk4P6S77aVDHIh9-leAgIyScfmpXqhj9d3UDj4UCz3wgy7uczqPBsadasb1nXkAnzXE4gY4VtJe8aCgYKAR4SARESFQHGX2MibS54WHYZ2-QB7nBrrtGwtg0178

But when I use your script to get an access token, it retrieves a much longer token which doesn't work when accessing my registry:
ya29.c.c0AY_VpZiDwXg2Rh8co06iVcLjYS463stzOHTHTj3IoSGQDzvTeDAjJuxkghYz_Ss79b5T5y2IN4l1fiBsW6OW0AEThO3AH52Za8N2EWDsCOXgBQ6qjcODspPZsWHfiujPazJgCVcOJuGkVWVQwvryRelgopJyBdH_knOJ4CyVx07pf_trrmW_Nx6GSCRtzNrNmkUNOnTvoYUwUTqBi4oFgtUStqL05cYbHEWrtZ9Z3f1YjnDmYX-lrcwWLbEIUI4gwhBS2W908Ur9pClsacGkREK77NaPK3M_J_7shc1YUtoAITBbk1gL45xnfUgLQEuvS4YhmpdNvrbxNPLJMxy2bQcUVvPaStDrwPoCnxyzZ1gFH9Ngl_a3yBeWIKT1C9u92oH396DB3wwsidJmkbIa17udSslUwV-v2Bo8qYIVjku1sziOJqhrQRhpJI5Rlr22rww-i8cOarqVFf7ntkQO7_66b_5BcUXB3d2zJ3VqcJv71byigsYZrc87bef96jzdX8ee30Iys_mtsJ0cVFiy9IwJZUBYXF1_zmbjb1q165zaROidX2wZlY0jpIxQVXqufWOyjbBO4d-ds81UVcJVkIsiUF0jBnckSXXqaZ9RZ9u24uQjbijz7XtmB7peuFv9YzQ50fqm_xljxJri-yws8BouhFxoetzs56aziw4cM8lhAZoh7kOtj7Fmju9hZXp1-WRleyVZbVRVXQcJk6qR7j3m1dS-fv1hF-gzVcsW97-tSajuh8UbSFv7elxh7Rq0MY9I0sWSqBnXisYJwB2uxndfawj7ahdYOXXcY5wov3IQsZM7--9RyWYW9vtmBah1aoMy71QFvpZIc-5xX3aOJmp_Fo9B6_khZyOkVSkxp8m-UZ1elceBzXsipZXU3l-XXYp5QJi8zf6tp2q8-e70hkmwz89IBXJ8SkjXlsZxOFMUrYWu2o1ZBB9O53ywxrqX8F9Jr1Sc7sVcy08zbjFZmJlbX76cF2k1fSw53jj2eoqivUMMVo7n

I've changed these tokens so they won't work ;) But does anyone know why the shorter token would work, but the longer one would give a permission issue??

[cyberslot]

@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue:
denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist)
When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to.
I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one.
Do I need to mount a volume as mentioned here by @bkanuk: #319 (comment) ?

Figured this out - I had to delete the whole deployment and not just the pod for this to show in the containers file system. However, still getting a permission error when trying to access my Artifact Registry repository. Interestingly, when I run a "gcloud auth print-access-token" command and save the token details, it looks like this (much shorter token and this works when I try to access my registry) ya29.a0AfB_byA_q-IsTFhKXdjs32Q3vMVsQ6v1jw1u0BHr7g9Rz_YrEUt4BSJ6Ssvo3jHKd83ld0Utqrk9zLBSahL2G9KBk4P6S77aVDHIh9-leAgIyScfmpXqhj9d3UDj4UCz3wgy7uczqPBsadasb1nXkAnzXE4gY4VtJe8aCgYKAR4SARESFQHGX2MibS54WHYZ2-QB7nBrrtGwtg0178

But when I use your script to get an access token, it retrieves a much longer token which doesn't work when accessing my registry: ya29.c.c0AY_VpZiDwXg2Rh8co06iVcLjYS463stzOHTHTj3IoSGQDzvTeDAjJuxkghYz_Ss79b5T5y2IN4l1fiBsW6OW0AEThO3AH52Za8N2EWDsCOXgBQ6qjcODspPZsWHfiujPazJgCVcOJuGkVWVQwvryRelgopJyBdH_knOJ4CyVx07pf_trrmW_Nx6GSCRtzNrNmkUNOnTvoYUwUTqBi4oFgtUStqL05cYbHEWrtZ9Z3f1YjnDmYX-lrcwWLbEIUI4gwhBS2W908Ur9pClsacGkREK77NaPK3M_J_7shc1YUtoAITBbk1gL45xnfUgLQEuvS4YhmpdNvrbxNPLJMxy2bQcUVvPaStDrwPoCnxyzZ1gFH9Ngl_a3yBeWIKT1C9u92oH396DB3wwsidJmkbIa17udSslUwV-v2Bo8qYIVjku1sziOJqhrQRhpJI5Rlr22rww-i8cOarqVFf7ntkQO7_66b_5BcUXB3d2zJ3VqcJv71byigsYZrc87bef96jzdX8ee30Iys_mtsJ0cVFiy9IwJZUBYXF1_zmbjb1q165zaROidX2wZlY0jpIxQVXqufWOyjbBO4d-ds81UVcJVkIsiUF0jBnckSXXqaZ9RZ9u24uQjbijz7XtmB7peuFv9YzQ50fqm_xljxJri-yws8BouhFxoetzs56aziw4cM8lhAZoh7kOtj7Fmju9hZXp1-WRleyVZbVRVXQcJk6qR7j3m1dS-fv1hF-gzVcsW97-tSajuh8UbSFv7elxh7Rq0MY9I0sWSqBnXisYJwB2uxndfawj7ahdYOXXcY5wov3IQsZM7--9RyWYW9vtmBah1aoMy71QFvpZIc-5xX3aOJmp_Fo9B6_khZyOkVSkxp8m-UZ1elceBzXsipZXU3l-XXYp5QJi8zf6tp2q8-e70hkmwz89IBXJ8SkjXlsZxOFMUrYWu2o1ZBB9O53ywxrqX8F9Jr1Sc7sVcy08zbjFZmJlbX76cF2k1fSw53jj2eoqivUMMVo7n

I've changed these tokens so they won't work ;) But does anyone know why the shorter token would work, but the longer one would give a permission issue??

In the script, the command used for token invocation (wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token) is exploiting an API call. It achieves two things:

• leverages the Google Compute Engine Metadata Server API. This API provides access to instance-specific information like project ID, zone, and service accounts.
• retrieves a short-lived access token associated with the default service account on the Google Compute Engine (GCE) instance.

On the other hand "gcloud auth print-access-token" command doesn't directly interact with an API endpoint. It utilizes the Google Cloud SDK libraries installed on your local machine to fetch and print an access token. So In essence, there's no single API endpoint involved. The gcloud tool and the local SDK libraries handle the entire process of fetching and printing the access token using established Google Cloud authentication mechanisms.

I hope this sheds some light on the matter.

[cooervo]

Example for multiregion us

  registries: 
    - name: GCP Artifact Registry
      api_url: https://us-docker.pkg.dev
      prefix: us-docker.pkg.dev
      credentials: ext:/scripts/gcp-auth.sh # defined in authScripts below
      credsexpire: 30m
      default: true

...

authScripts:
  enabled: true
  scripts:
    gcp-auth.sh: |
      #!/bin/sh
      ACCESS_TOKEN=$(wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -q -O - | grep -Eo '"access_token":.*?[^\\]",' | cut -d '"' -f 4)
      echo "oauth2accesstoken:$ACCESS_TOKEN"

...

# Apart from the above don't forget to bind the GCP service account to the K8s service account in `values.yaml`:
serviceAccount:
  create: true
  annotations: {
    # the GCP service account to bind the present k8s service account
    iam.gke.io/gcp-service-account: sa-gcp@my-project.iam.gserviceaccount.com
  }

I wrote an article here with more instructions in case someone needs further troubleshooting, I hope it saves you time since I didn't find a comprehensive resource for authenticating argocd-image-updater in GCP Artifact Registry.