Cannot pull images from Azure Container Registry #550

matteotumiati · 2023-04-03T11:22:17Z

Describe the bug
I cannot pull images that are stored in Azure Container Registry.

To Reproduce

Install ArgoCD from scratch
Install argocd-image-updater (kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml)
Create a Kubernetes secret with credentials for the ACR (access using admin):

kubectl create secret docker-registry acr \
    --namespace argocd \
    --docker-server=<container-registry-name>.azurecr.io \
    --docker-username=<container-registry-name> \
    --docker-password=<password>

Edit the ConfigMap argocd-image-updater-config (kubectl edit configmap argocd-image-updater-config -n argocd) to include the section for custom registries:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-image-updater-config","app.kubernetes.io/part-of":"argocd-image-updater"},"name":"argocd-image-updater-config","namespace":"argocd"}}
  creationTimestamp: "2023-04-03T08:12:02Z"
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
  namespace: argocd
  resourceVersion: "13173"
  uid: 99218d75-12cf-41cf-b9e4-198fce05d60a
data:
  registries.conf: |
    registries:
    - name: 'azure'
      prefix: <container-registry-name>.azurecr.io
      api_url: https://<container-registry-name>.azurecr.io/
      credentials: pullsecret:argocd/acr
      default: true

It is defined as pullsecret, because the secret created in step 3 has a field .dockerconfigjson in the .data section.

Restart the deployment, to make sure the ConfigMap is reloaded (kubectl -n argocd rollout restart deployment argocd-image-updater)
Create an empty Helm chart and an application to deploy a simple container from the container registry in Azure. It will fail with Init:ImagePullBackOff.

Expected behavior
The image can be pulled successfully.

Additional context
Passing credentials to the CLI test command works fine.

argocd-image-updater test <container-registry-name>.azurecr.io/<image>:<tag> --credentials pullsecret:argocd/acr

Version
Latest

Logs

argocd-image-updater test <container-registry-name>.azurecr.io/busybox:1.36

time="2023-04-03T15:06:26+02:00" level=debug msg="Creating in-cluster Kubernetes client"
time="2023-04-03T15:06:26+02:00" level=info msg="retrieving information about image" image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io
time="2023-04-03T15:06:26+02:00" level=debug msg="setting rate limit to 20 requests per second" prefix=<container-registry-name>.azurecr.io registry="https://<container-registry-name>.azurecr.io"
time="2023-04-03T15:06:26+02:00" level=debug msg="Inferred registry from prefix <container-registry-name>.azurecr.io to use API https://<container-registry-name>.azurecr.io"
time="2023-04-03T15:06:26+02:00" level=info msg="Fetching available tags and metadata from registry" application=test image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io
time="2023-04-03T15:06:26+02:00" level=fatal msg="could not get tags: Get \"https://<container-registry-name>.azurecr.io/v2/busybox/tags/list\": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information." application=test image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io

The text was updated successfully, but these errors were encountered:

pa-mc · 2023-05-23T18:41:09Z

I can confirm. I have the exact same setup. Got the same error. Seem to only work when specifiying --credentials through subcommand test within the CLI.

/ $ cat /app/config/registries.conf
registries:
- name: ACR paservices
  prefix: azurecr.io
  api_url: https://paservices.azurecr.io
  credentials: pullsecret:argocd/pa-paservices-acr
  default: true

joeferreirapremera · 2023-06-29T22:51:30Z

have you tried to repo this with a kubelet identity that has Acrpull against the ACR?

Install azure-cli in Docker image in order to use the `az acr login` command. Can be used with Azure Managed Identities with the following script: ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config namespace: argocd data: log.level: debug registries.conf: | registries: - name: acrexample api_url: https://acrexample.azurecr.io/ prefix: acrexample.azurecr.io ping: yes insecure: no credentials: ext:/app/scripts/acr-login.sh credsexpire: 10h --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config-acr namespace: argocd data: acr-login.sh: | #!/bin/sh LOGIN=$(az login --identity) REGISTRY="acrexample" TOKEN=$(az acr login --name $REGISTRY --expose-token --output tsv --query accessToken) echo "00000000-0000-0000-0000-000000000000:$TOKEN" ``` Closes argoproj-labs#550 and argoproj-labs#473

Install azure-cli in Docker image in order to use the `az acr login` command. Can be used with Azure Managed Identities with the following script: ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config namespace: argocd data: log.level: debug registries.conf: | registries: - name: acrexample api_url: https://acrexample.azurecr.io/ prefix: acrexample.azurecr.io ping: yes insecure: no credentials: ext:/app/scripts/acr-login.sh credsexpire: 10h --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config-acr namespace: argocd data: acr-login.sh: | #!/bin/sh LOGIN=$(az login --identity) REGISTRY="acrexample" TOKEN=$(az acr login --name $REGISTRY --expose-token --output tsv --query accessToken) echo "00000000-0000-0000-0000-000000000000:$TOKEN" ``` Closes argoproj-labs#550 and argoproj-labs#473 Signed-off-by: Jarvis Yang <jarvis.yang@recurohealth.com>

Install azure-cli in Docker image in order to use the `az acr login` command. Can be used with Azure Managed Identities with the following script: ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config namespace: argocd data: log.level: debug registries.conf: | registries: - name: acrexample api_url: https://acrexample.azurecr.io/ prefix: acrexample.azurecr.io ping: yes insecure: no credentials: ext:/app/scripts/acr-login.sh credsexpire: 10h --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config-acr namespace: argocd data: acr-login.sh: | #!/bin/sh LOGIN=$(az login --identity) REGISTRY="acrexample" TOKEN=$(az acr login --name $REGISTRY --expose-token --output tsv --query accessToken) echo "00000000-0000-0000-0000-000000000000:$TOKEN" ``` Closes argoproj-labs#550 and argoproj-labs#473

Install azure-cli in Docker image in order to use the `az acr login` command. Can be used with Azure Managed Identities with the following script: ```yaml --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config namespace: argocd data: log.level: debug registries.conf: | registries: - name: acrexample api_url: https://acrexample.azurecr.io/ prefix: acrexample.azurecr.io ping: yes insecure: no credentials: ext:/app/scripts/acr-login.sh credsexpire: 10h --- apiVersion: v1 kind: ConfigMap metadata: name: argocd-image-updater-config-acr namespace: argocd data: acr-login.sh: | #!/bin/sh LOGIN=$(az login --identity) REGISTRY="acrexample" TOKEN=$(az acr login --name $REGISTRY --expose-token --output tsv --query accessToken) echo "00000000-0000-0000-0000-000000000000:$TOKEN" ``` Closes argoproj-labs#550 and argoproj-labs#473 Signed-off-by: Francesc Arbona <francesc.arbona@global.ntt>

Popeye4242 · 2024-02-09T07:29:03Z

Azure Container Registry currently works out of the box for azure kubernetes services. You just have to enable managed identity. Using azurecr when not using aks is probably not a good decision anyway. There is no need to specify any credentials if you are using managed identities.

https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer
https://learn.microsoft.com/en-us/azure/aks/use-managed-identity

Pionerd · 2024-04-10T15:19:17Z

First of all, if you force the pullSecret using the annotation, it also works. I consider this a workaround.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: test=<name>.azurecr.io/<image>
    argocd-image-updater.argoproj.io/test.pull-secret: pullsecret:argocd/acr-secret

Secondly, in my humble opinion importing a script to perform all actions necessary for Workload Identity is far from "just have to enable managed identity". I'm a huge fan of Workload Identity and use it in almost all components, but in none of them such a script addition is necessary.

Next to that, it just seems off that even though the credentials are correctly configured, they are only used when forced through the annotation. To me this is simply a bug.

Edit (12-Apr):
So after testing with Workload Identity (@etiennetremel Thanks for the docs) I noticed that exactly the same issue as with the pull secret is going on:

  $ argocd-image-updater test <acr_name>.azurecr.io/<image_name>@sha256:<image_sha>
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=<acr_name>.azurecr.io registry="https://<acr_name>.azurecr.io"
DEBU[0000] Inferred registry from prefix <acr_name>.azurecr.io to use API https://<acr_name>.azurecr.io
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
FATA[0000] could not get tags: Get "https://<acr_name>.azurecr.io/v2/<image_name>/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
/ $ cat /app/config/registries.conf
registries:
  - api_url: https://<acr_name>.azurecr.io
    credentials: ext:/scripts/auth.sh
    credsexpire: 1h
    default: true
    name: <acr_name>
    prefix: <acr_name>.azurecr.io
/ $ argocd-image-updater test <acr_name>.azurecr.io/<image_name>@sha256:<image_sha> --credentials ext:/scripts/auth.sh
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=<acr_name>.azurecr.io registry="https://<acr_name>.azurecr.io"
DEBU[0000] Inferred registry from prefix <acr_name>.azurecr.io to use API https://<acr_name>.azurecr.io
INFO[0000] /scripts/auth.sh                              dir= execID=8aa46
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
INFO[0000] Found 2 tags in registry                      application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] could not parse input tag dev as semver: Invalid Semantic Version
DEBU[0000] could not parse input tag prd as semver: Invalid Semantic Version
INFO[0000] latest image according to constraint is <acr_name>.azurecr.io/<image_name>@sha256:<image_sha>  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io

So basically, the issue remains: the config in the registries.conf is not picked up. If the same credentials are forced (either through the --credentials flag when using argocd-image-updater test or the annotation) everything works.

mtrin · 2024-06-07T05:15:47Z

Stumbled on this by chance. I remember taking a while to figure out but I made it work with acr without workload ids.

  registries.conf: |
    registries:
    - name: <acrname>
      api_url: https://<acrname>.azurecr.io
      ping: no
      credentials: secret:argocd/<secretname>#creds   
      prefix:  <acrname>.azurecr.io

notice how the secret key is specified with the #creds suffix
so the secret should be like

data:
  creds: clientid:secret

notice the creds format is important too

Hope it helps someone. This would be good to clarify in the docs @argoproj-labs

matteotumiati added the bug Something isn't working label Apr 3, 2023

xescab linked a pull request Jul 4, 2023 that will close this issue

feat: Allow ACR authentication using Azure CLI #586

Open

jwhy89 mentioned this issue Aug 17, 2023

feat: Allow ACR authentication using Azure CLI #609

Closed

etiennetremel mentioned this issue Feb 17, 2024

docs: add instructions to authenticate to Azure Container Registry with workload identity #676

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot pull images from Azure Container Registry #550

Cannot pull images from Azure Container Registry #550

matteotumiati commented Apr 3, 2023 •

edited

pa-mc commented May 23, 2023

joeferreirapremera commented Jun 29, 2023

Popeye4242 commented Feb 9, 2024

Pionerd commented Apr 10, 2024 •

edited

mtrin commented Jun 7, 2024

Cannot pull images from Azure Container Registry #550

Cannot pull images from Azure Container Registry #550

Comments

matteotumiati commented Apr 3, 2023 • edited

pa-mc commented May 23, 2023

joeferreirapremera commented Jun 29, 2023

Popeye4242 commented Feb 9, 2024

Pionerd commented Apr 10, 2024 • edited

mtrin commented Jun 7, 2024

matteotumiati commented Apr 3, 2023 •

edited

Pionerd commented Apr 10, 2024 •

edited