External k8s Cluster unable to pull from ACR - Unauthorized: Authentication required

[mmisztal1980]

What happened:
I have a PoC environment, outside of Azure which i use for experimentation, the env. has been running for a couple of weeks without any issues. For the past couple of days I've started to observe that the cluster is incapable of pulling images hosted in my ACR registry: cloudtechnologies.azurecr.io

Upon examination, all of the pods report Error: ImagePullBackOff

There is also message stating:

Failed to pull image "cloudtechnologies.azurecr.io/poc-k8s-web:20181007.1": rpc error: code = Unknown desc = Error response from daemon: Get https://cloudtechnologies.azurecr.io/v2/poc-k8s-web/manifests/20181007.1: unauthorized: authentication required

What you expected to happen:
I expected to the pod to be able to pull the container image successfully

How to reproduce it (as minimally and precisely as possible):
The cluster uses the admin user's credentials and has had a secret created, in the default namespace, with:

kubectl create secret docker-registry cloud-technologies-registry \
    --docker-server=cloudtechnologies.azurecr.io \
    --docker-username=cloudtechnologies \
    --docker-password=<PRIMARY ACCESS KEY> \
    --docker-email=<MY EMAIL>

The deployment, in the default namespace, which is attempting to pull the image has the following fragment defined:

spec:
    imagePullSecrets:
    - cloud-technologies-registry
    containers:
    - name: poc-k8s-web
      image: cloudtechnologies.azurecr.io/poc-k8s-web:20181007.1
      imagePullPolicy: Always
      env:
      - name: 'ASPNETCORE_ENVIRONMENT'
        value: 'PRODUCTION'
     ports:
     - containerPort: 80

Anything else we need to know?:

• My cluster is NOT hosted on Azure, so I'm not using az aks cli etc.
• I've recently reprovisioned the ACR however got the same result
• Before reprovisioning the ACR, the password was changed a few times and the secret was refreshed cluster-side
• I've tried a docker login locally and I was able to login successfully
• I'm unable to ping ACR's endpoint neither from my local machine, nor from any of the k8s nodes

Environment:

• Kubernetes version (use kubectl version): 1.9.0
• Size of cluster (how many worker nodes are in the cluster?) 3
• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.) HTTP microservices
• Others:

[andyzhangx]

It looks like your provided docker-password is wrong, I tried on k8s v1.12 with secret, it works well, see example

[mmisztal1980]

I copied it from the portal using the copy button, I'll try to copy manually next time

[andyzhangx]

you may also pull image manually on one node to make sure the connection is ok:
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-get-started-azure-cli

[mmisztal1980]

@andyzhangx have you tried it against my ACR instance? I've tried both access keys, copied them using both methods (copy button and ctrl+c). I don't see anything wrong with my process so I'm wondering what may be wrong.

[mmisztal1980]

Moving this to the ACR repo, sorry for realizing late that this is not the proper repo for this discussion.

[mmisztal1980]

Issue moved here: Azure/acr#138