AKS-ACR unauthorized: authentication required #35411

sameer-kumar · 2019-07-18T14:34:35Z

I'm having issues with a pod in AKS pulling image from ACR.
I tried the steps documented here https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks#grant-aks-access-to-acr but no luck.

The error I received is :

Failed to pull image "test.azurecr.io/q/p:01": rpc error: code = Unknown desc = Error response from daemon: Get https://test.azurecr.io/v2/q/p/manifests/01: unauthorized: authentication required

I'm on AKS 1.14.0 and trying this on Windows Node Pool.

What am I missing here?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ed59d472-ac33-5c03-dbb7-0e86e88c7dd8
Version Independent ID: 69e1ad8c-2dd3-cc1b-2b31-996a5d866cc0
Content: Authenticate with Azure Container Registry from Azure Kubernetes Service
Content Source: articles/container-registry/container-registry-auth-aks.md
Service: container-service
GitHub Login: @dlepow
Microsoft Alias: danlep

mimckitt · 2019-07-18T17:13:02Z

Thanks for the feedback! We are currently investigating and will update you shortly.

mimckitt · 2019-07-18T21:11:40Z

@sameer-kumar are you having any issues logging into your ACR or is it just with pulling images?

meidal · 2019-07-19T10:27:45Z

Hi @MicahMcKittrick-MSFT,

I'm having the same problem. This has been working until today without issues and now it seems the service principal that's configured on our AKS has dropped it's authentication for some reason and can no longer pull images from our ACR

meidal · 2019-07-19T10:30:48Z

I can login and push images to the ACR, our AKS's service principal seems to be the problem

sameer-kumar · 2019-07-19T11:59:14Z

@MicahMcKittrick-MSFT same with me. I verified that I can log in to ACR using admin credentials as well as with a custom SPN creds from a windows node in the same subnet as AKS cluster. However, AKS cluster SPN is unable to authenticate and hence can't pull images.

jszanto · 2019-07-20T21:35:19Z

I have the same issue, I am using a service principle for authentication, error can be reproduced with the following script:

ACR_ID=$(az acr show -n $myacr -g $myresourcegroup --query "id" -o tsv)
registryPassword=$(az ad sp create-for-rbac -n $myacr-push --scopes $ACR_ID --role acrpush --query password -o tsv)
registryUsername=$(az ad sp show --id http://$myacr-push --query appId -o tsv)

docker login --username $registryUsername --password $registryPassword "$myacr.azurecr.io"

Gives the following response

Error response from daemon: Get https://$myacr.azurecr.io/v2/: unauthorized: authentication required

UPDATE: After running this about 5 times it randomly seems to work. Another thing I noticed is that after creating a SP it takes about 30-60s before it become active. So you can't immediately use if for instance in a deployment script.

meidal · 2019-07-20T22:20:38Z

Hey guys, managed to get this working. 1) confirm your aks spn has acrpull role on your acr. 2) confirm the repo/image really exists on your acr, I had this error and the repo/image didn't exist

mimckitt · 2019-07-22T22:35:06Z

Thanks for the update all! Apologies for the delay in response. I was out for the weekend.

Appears the issue is no longer present. Is that correct?

sameer-kumar · 2019-07-22T23:57:37Z

Issue still exists. Same error with AKS spn. However, no error with custom spn.

…

On Mon, Jul 22, 2019, 6:35 PM Micah ***@***.***> wrote: Thanks for the update all! Apologies for the delay in response. I was out for the weekend. Appears the issue is no longer present. Is that correct? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35411?email_source=notifications&email_token=AA35XPIPGZ5K23GDPZ6UUY3QAYY3DA5CNFSM4IE4BAF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2RLROA#issuecomment-513980600>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA35XPPINNJLO645BF76TGDQAYY3DANCNFSM4IE4BAFQ> .

jszanto · 2019-07-23T12:46:24Z

For now I am not having any issues, however the bug in which you get an unauthorized response whenever your image doesn't exist is kind of annoying. It makes debugging things quite difficult.

mimckitt · 2019-07-23T17:05:53Z

I am not seeing any issues at this time.

For anyone that is, feel free to reach out to me via AzCommunity@Microsoft.com and provide me with your Azure SubscriptionID and link to this issue. I can then enable you for a free support request to have this looked into further.

MinghuaJiang · 2019-07-26T22:15:25Z

I'm able to access acr from aks if I do kubectl apply after following the guide, but if I do a kubectl set image to update the image, it returns unauthorized when acrpull like what was mentioned above.

MinghuaJiang · 2019-07-26T22:16:06Z

@MicahMcKittrick-MSFT any idea on it?

mimckitt · 2019-07-26T22:46:54Z

@sameer-kumar I am not seeing anything in this doc related to kubectl set image. Are you following another doc by chance?

MinghuaJiang · 2019-07-26T22:59:57Z

the doc itself doesn't include set image, I just do need to run the set image command to update image, and not sure why it doesn't work since my kubenetes cluster should already have access to pull acr.

mimckitt · 2019-07-26T23:15:12Z

What is the command you are using? It should be something like

kubectl set image deployment azure-vote-front azure-vote-front=<acrLoginServer>/azure-vote-front:v2

MinghuaJiang · 2019-07-26T23:49:49Z

@MicahMcKittrick-MSFT Yes, this is exactly the command I used. I just want to update the docker image being deployed from say v1 to v2, but it says unauthorized. The first deploy with kubectl apply works well for me.

mimckitt · 2019-07-26T23:52:15Z

@sameer-kumar try this doc to see if you can do all the steps

https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-app-update

If not, freel free to email me at AzCommunity@microsoft.com as well with your SubscriptionID and link to this thread and I can get you in contact with Technical Support.

MinghuaJiang · 2019-07-26T23:58:58Z

@MicahMcKittrick-MSFT that's exactly the doc I reference to.

mimckitt · 2019-07-27T00:00:58Z

@sameer-kumar go ahead and shoot me that email with the details and we can get it sorted out :)

oaicstef · 2019-11-10T12:11:18Z

Hi,
do you have a solution for this issue, it is happening to me, too.
I'm using the aks preview feature with windows node pool.
Using AKS 1.14.8 with a private Azure container registry,
the kubernetes pod is not able to pull the image, " unauthorized: authentication required".

Please, if there is another thread to follow, could you point me to it?
Cheers.

junsaw · 2019-11-19T02:52:42Z

Specify your cabernets secret name in the yaml configuration:

e.g:
apiVersion: v1
kind: Pod
metadata:
name: private-reg
spec:
containers:

name: private-reg-container
image:
imagePullSecrets:
name: regcred

See reference URL:

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

junsaw · 2019-11-19T02:54:35Z

                                             /\
                                              |

Specify you container secret ---

StingyJack · 2020-01-01T19:26:55Z

Using docker pull <image/tag> works just fine after running az login and then sudo az acr login --name <az container reg name>, but if i use docker-compose (which is kind of needed for a multi container app) I get this error.

This is only non functional so far on our new testing/staging environment (an Azure hosted Ubuntu VM). It works fine from my windows pc and other developers workstations (win, osx), and other environments (an AWS hosted Ubuntu vm that is the current testing/staging environment) can docker-compose pull images from the same container registry using the same docker-compose.yml file.

$> az login

{lists my subscriptions, the default subscription has the container registry in it}

$> sudo az acr login --name myconreg01

Login Succeeded

$> sudo docker-compose --verbose -f docker/my.docker-compose.yml pull

compose.config.config.find: Using configuration files: ./docker/my.docker-compose.yml
WARNING: compose.config.environment.__getitem__: The INSIGHTOPS_REGION variable is not set. Defaulting to a blank string.
WARNING: compose.config.environment.__getitem__: The INSIGHTOPS_TOKEN variable is not set. Defaulting to a blank string.
docker.auth.auth.load_config: Found 'auths' section
docker.auth.auth.parse_auth: Found entry (registry=u'https://index.docker.io/v1/', username=u'myUserName')
docker.auth.auth.parse_auth: Found entry (registry=u'myconreg01.azurecr.io', username=u'00000000-0000-0000-0000-000000000000')
docker.auth.auth.load_config: Found 'HttpHeaders' section
compose.cli.command.get_client: docker-compose version 1.8.0, build unknown
docker-py version: 1.9.0
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
compose.cli.command.get_client: Docker base_url: http+docker://localunixsocket
compose.cli.command.get_client: Docker version: KernelVersion=4.15.0-1063-azure, Components=[{u'Version': u'18.09.7', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.15.0-1063-azure'... (truncated)
compose.service.pull: Pulling beat (myconreg01.azurecr.io/primary-worker-image:latest)...
compose.cli.verbose_proxy.proxy_callable: docker pull <- (u'myconreg01.azurecr.io/primary-worker-image', tag=u'latest', stream=True)
docker.api.image.pull: Looking for auth config
docker.auth.auth.resolve_authconfig: Looking for auth entry for u'myconreg01.azurecr.io'
docker.auth.auth.resolve_authconfig: Found u'myconreg01.azurecr.io'
docker.api.image.pull: Found auth config
ERROR: compose.cli.errors.log_api_error: Get https://myconreg01.azurecr.io/v2/primary-worker-image/manifests/latest: unauthorized: authentication required

jszanto · 2020-01-01T20:14:07Z

@StingyJack You are running your docker-compose command as root (why?), while you run your az acr login command as another user. This command stores the docker registry credentials somewhere in the home directory of your user (~/.docker), therefore they can not be found by the root user.

StingyJack · 2020-01-02T03:47:26Z

@jszanto - I'm not running anything as root, they are all running as the "my-admin" account. Some are elevated, but still running as the "my-admin" account. Needing sudo for this seems to be how docker works "out of the box". I've had quite enough weird, frustrating, and time consuming trouble with docker (both the program and the org) over the last year or two that I'm not terribly interested in investing time into researching and attempting better configurations for it, I need it to work consistently first.

I see the confusion point I created though, the az acr login command should have had sudo in front of it because it doesnt have access to read the config.json without it. I've corrected that typo, sorry about that.

For completeness this is the error I get when I forget to sudo the az acr login or the docker-compose commands.

$ az acr login --name myconreg01

An error occurred: DOCKER_COMMAND_ERROR
WARNING: Error loading config file: /home/my-admin/.docker/config.json: stat /home/my-admin/.docker/config.json: permission denied
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.39/containers/json: dial unix /var/run/docker.sock: connect: permission denied

This same command set works in other like environments so I dont think I need to change them. It would be nice to know what else may be needed in the environment to be able to connect to ACR.

EDIT: This was due to docker-compose being at an old version, like 1.8 or something like that. Updating to 1.24 fixed the problem.

pbrit · 2020-01-12T13:22:38Z

@MicahMcKittrick-MSFT The issue can be easily reproduced on AKS 1.15.x. Could you please re-open it?

johnwrobinson · 2020-01-20T17:37:11Z

Here to confirm this is still happening in 1.15.7...

pbrit · 2020-01-20T17:54:59Z

@MicahMcKittrick-MSFT The issue can be easily reproduced on AKS 1.15.x. Could you please re-open it?

Please disregard this. It turned out the tag in question didn't exist. After I fixed the name of the tag It started to work as intended. @johnwrobinson, this might happen to you as well.

unauthorized: authentication required for an absent tag is quite confusing though.

StingyJack · 2020-01-20T23:39:44Z

Bad (unhelpful, uninformative, or just wrong) error messages should be nontrivial deductions from the offenders compensation.

Sudharma · 2020-01-21T10:11:35Z

still this does not work for us. I mean the behaviour is inconsistent. It works sometimes and after few deployments it fails and if we start the deployment again. it works. No idea whats wrong here.

The original SP is with Contributor role and the same SPN has the AcrPullRole. If I list the role assignment with --all then I can see the role

colin-lyman · 2020-01-29T11:35:05Z

From a previous comment earlier in the thread. My issue came from the tag of the image not being correct. Worth double-checking the repo and image tag exists.

MartinKosicky · 2020-02-17T06:57:02Z

actually I can login to az login using SP credentials and az acr login success. However the aks can't seem to pull the images

colin-lyman · 2020-02-17T08:15:47Z

@MartinKosicky - check the SP has the AcrPullRole permission.

MartinKosicky · 2020-02-17T08:22:36Z

@colin-lyman actually I rememberd that I accidentally last week updated aks with a different SP (using terraform). I right away repaired it, and also "aks show" told me that it is correct. However it didn't work. Deleting and recreating the whole AKS helped here. It seems that updating the service principal on aks can really break some things.

audunsol · 2020-03-02T13:35:37Z

I had the same problem now. I verified that the image tag was correct by pulling it on my local machine without problems.
I had scripted the process for granting aks pull access to acr, something copy-pasted from some Microsoft documentation at some point (unfortunately I did not save the link):

# Get the id of the service principal configured for AKS
CLIENT_ID=$(az aks show --resource-group $AKS_RESOURCE_GROUP --name $AKS_CLUSTER_NAME --query "servicePrincipalProfile.clientId" --output tsv)

# Get the ACR registry resource id
ACR_ID=$(az acr show --name $ACR_NAME --resource-group $ACR_RESOURCE_GROUP --query "id" --output tsv)

# Create role assignment
az role assignment create --assignee $CLIENT_ID --role acrpull --scope $ACR_ID

When trying to find the docs back again now, I found this new command (to me at least) here:
https://docs.microsoft.com/en-us/azure/aks/cluster-container-registry-integration

So, I tried that new command instead:

az aks update -n $AKS_CLUSTER_NAME -g $AKS_RESOURCE_GROUP --attach-acr $ACR_NAME

I had to upgrade my azure cli from v 2.0.61 to 2.1.0 to get the update operation available, and got this error after a while:

AAD role propagation done[############################################]  100.0000%request failed: Error occurred in request., RetryError: HTTPSConnectionPool(host='management.azure.com', port=443): Max retries exceeded with url: /subscriptions/<subscription-GUID>/resourceGroups/<myAKS-rgName>/providers/Microsoft.ContainerService/managedClusters/<myClusterName>?api-version=2019-11-01 (Caused by ResponseError('too many 500 error responses',))

...but after deleting my pod and have the deployment spin up a new one after this (no config changes), it was finally able to pull the image.

Not sure when and how this changed. I have pulled images successfully without changing this setup for months, but now all of a sudden it didn't work any more. Hope this helps someone else.

UPDATE:
It seems I was a bit too quick on my conclusion. It only worked transiently the first time after the above command. Transient absent of error, then. Quite frustrating.

audunsol · 2020-03-02T21:59:46Z

OK. Found my error: The service principal had expired 🤦‍♂ (I wasn't aware/paying attention of the one year expiration time). Now that I know, I can also see it from the logs on the AKS resource in the Azure Portal.

To fix it, I followed the instructions here https://docs.microsoft.com/en-us/azure/aks/update-credentials

Sorry about the noise, and hope this may help someone else landing on this page.

matthklo · 2020-04-20T08:44:06Z

OK. Found my error: The service principal had expired 🤦‍♂ (I wasn't aware/paying attention of the one year expiration time). Now that I know, I can also see it from the logs on the AKS resource in the Azure Portal.

To fix it, I followed the instructions here https://docs.microsoft.com/en-us/azure/aks/update-credentials

Sorry about the noise, and hope this may help someone else landing on this page.

Thanks Pal! You really save my day! That is exactly the same error for my case.

ermancom · 2021-01-07T23:52:45Z

I'm having issues with a pod in AKS pulling image from ACR.
I tried the steps documented here https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks#grant-aks-access-to-acr but no luck.

The error I received is :

Failed to pull image "test.azurecr.io/q/p:01": rpc error: code = Unknown desc = Error response from daemon: Get https://test.azurecr.io/v2/q/p/manifests/01: unauthorized: authentication required

I'm on AKS 1.14.0 and trying this on Windows Node Pool.

What am I missing here?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ed59d472-ac33-5c03-dbb7-0e86e88c7dd8

Version Independent ID: 69e1ad8c-2dd3-cc1b-2b31-996a5d866cc0

Content: Authenticate with Azure Container Registry from Azure Kubernetes Service

Content Source: articles/container-registry/container-registry-auth-aks.md

Service: container-service

GitHub Login: @dlepow

Microsoft Alias: danlep

I had a same problem and I solved by attaching ACR to AKS . I applied the commands below.
az aks update -n myAKSCluster -g myResourceGroup --attach-acr
If the more information you can look at the https://docs.microsoft.com/en-us/azure/aks/cluster-container-registry-integration#grant-aks-access-to-acr

brandonmaul · 2022-02-06T16:45:31Z

This was my problem as well! 2 hours trying to find a solution because I thought the aks/acr connection was incorrect.
Turns out, the image with the tag simply wasn't found.

PRMerger14 added the Pri3 label Jul 18, 2019

mimckitt self-assigned this Jul 18, 2019

mimckitt added container-service/svc cxp product-question triaged labels Jul 18, 2019

mimckitt added in-progress needs-more-info labels Jul 18, 2019

mimckitt removed the needs-more-info label Jul 22, 2019

mimckitt closed this as completed Jul 23, 2019

AKS-ACR unauthorized: authentication required #35411

AKS-ACR unauthorized: authentication required #35411

Comments

sameer-kumar commented Jul 18, 2019 • edited by mimckitt

Document Details

mimckitt commented Jul 18, 2019

mimckitt commented Jul 18, 2019

meidal commented Jul 19, 2019

meidal commented Jul 19, 2019

sameer-kumar commented Jul 19, 2019 • edited

jszanto commented Jul 20, 2019 • edited

meidal commented Jul 20, 2019

mimckitt commented Jul 22, 2019

sameer-kumar commented Jul 22, 2019 via email

jszanto commented Jul 23, 2019

mimckitt commented Jul 23, 2019

MinghuaJiang commented Jul 26, 2019

MinghuaJiang commented Jul 26, 2019

mimckitt commented Jul 26, 2019

MinghuaJiang commented Jul 26, 2019

mimckitt commented Jul 26, 2019

MinghuaJiang commented Jul 26, 2019

mimckitt commented Jul 26, 2019

MinghuaJiang commented Jul 26, 2019

mimckitt commented Jul 27, 2019

oaicstef commented Nov 10, 2019

junsaw commented Nov 19, 2019

junsaw commented Nov 19, 2019

StingyJack commented Jan 1, 2020 • edited

jszanto commented Jan 1, 2020 • edited

StingyJack commented Jan 2, 2020 • edited

pbrit commented Jan 12, 2020

johnwrobinson commented Jan 20, 2020

pbrit commented Jan 20, 2020

StingyJack commented Jan 20, 2020

Sudharma commented Jan 21, 2020

colin-lyman commented Jan 29, 2020

MartinKosicky commented Feb 17, 2020

colin-lyman commented Feb 17, 2020

MartinKosicky commented Feb 17, 2020

audunsol commented Mar 2, 2020 • edited

audunsol commented Mar 2, 2020

matthklo commented Apr 20, 2020 • edited

ermancom commented Jan 7, 2021

Document Details

brandonmaul commented Feb 6, 2022

sameer-kumar commented Jul 18, 2019 •

edited by mimckitt

sameer-kumar commented Jul 19, 2019 •

edited

jszanto commented Jul 20, 2019 •

edited

StingyJack commented Jan 1, 2020 •

edited

jszanto commented Jan 1, 2020 •

edited

StingyJack commented Jan 2, 2020 •

edited

audunsol commented Mar 2, 2020 •

edited

matthklo commented Apr 20, 2020 •

edited