az acr login with private endpoint goes via public route #17137

r3-jerrysteele · 2021-02-26T17:19:03Z

Describe the bug

Command Name
az acr login

Errors:

Error response from daemon: Get https://myregistryname.azurecr.io/v2/: denied: client with IP '<my public IP>' is not allowed access. Refer https://aka.ms/acr/firewall to grant access
Login failed.

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Connect to Azure VPN
az acr login -n myregistryname --subscription mysubscription

Expected Behavior

Login to the ACR's private endpoint via the connected VPN

Environment Summary

Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid
Python 3.6.10
Installer: DEB

azure-cli 2.19.1

Using WSL1

Additional Context

I am logged into a VPN (Azure Virtual Network Gateway), I have made the necessary adjustments to the /etc/resolv.conf so that the ACR resolves to the Private Link IP:

Server:         10.x.x.x.
Address:        10x.x.x#53

Non-authoritative answer:
myregistryname.azurecr.io canonical name = myregistryname.privatelink.azurecr.io.
Name:   myregistryname.privatelink.azurecr.io
Address: 10.x.x.x

The ACR is configured not to allow public access, but has a private endpoint configured which is known to work

I can curl the URL of the ACR and get a 200 response (as it goes via the private address), it seems that the issue is with azure-cli itself.

ip route 10.x.x.x also shows that the route goes via the VPN interface

The text was updated successfully, but these errors were encountered:

ghost · 2021-03-01T04:02:45Z

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @toddysm, @yugangw-msft.

Issue Details

Describe the bug

Command Name
az acr login

Errors:

Error response from daemon: Get https://myregistryname.azurecr.io/v2/: denied: client with IP '<my public IP>' is not allowed access. Refer https://aka.ms/acr/firewall to grant access
Login failed.

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Connect to Azure VPN
az acr login -n myregistryname --subscription mysubscription

Expected Behavior

Login to the ACR's private endpoint via the connected VPN

Environment Summary

Linux-4.4.0-19041-Microsoft-x86_64-with-debian-bullseye-sid
Python 3.6.10
Installer: DEB

azure-cli 2.19.1

Using WSL1

Additional Context

I am logged into a VPN (Azure Virtual Network Gateway), I have made the necessary adjustments to the /etc/resolv.conf so that the ACR resolves to the Private Link IP:

Server:         10.x.x.x.
Address:        10x.x.x#53

Non-authoritative answer:
myregistryname.azurecr.io canonical name = myregistryname.privatelink.azurecr.io.
Name:   myregistryname.privatelink.azurecr.io
Address: 10.x.x.x

The ACR is configured not to allow public access, but has a private endpoint configured which is known to work

I can curl the URL of the ACR and get a 200 response (as it goes via the private address), it seems that the issue is with azure-cli itself.

ip route 10.x.x.x also shows that the route goes via the VPN interface

Author:	r3-jerrysteele
Assignees:	-
Labels:	`Container Registry`, `Service Attention`, `needs-triage`, `question`
Milestone:	-

yungezz · 2021-03-01T04:02:51Z

route to appropriate team

yugangw-msft · 2021-03-25T02:37:45Z

@r3-jerrysteele, sorry for missing this. The best way to diagnose this is to verify the DNS setting. If not resolving to the private ip, then the error is expected. Doc is here:

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-private-link#validate-private-link-connection.

Please let us know what you found out

fieldp · 2021-11-18T13:55:46Z

Hi. I have the exact same issue. Not sure if it's a CLI issue or user error, so I posted it on the Microsoft Q&A forum. https://docs.microsoft.com/en-us/answers/questions/632408/trying-to-access-an-azure-container-registry-with.html

LahariChidura · 2022-05-16T15:50:44Z

I am facing the same issue.
The ACR DNS resolves to private IP address in the vnet -

however, az acr login throws this error -

fieldp · 2022-05-17T09:15:47Z

Good to know not just me then. I gave up. I had a back and forth with MS support but just could not solve it. If anyone has a new ideas I would be happy try :-)

…

On May 16, 2022, fieldp ***@***.***> wrote: I am facing the same issue. The ACR DNS resolves to private IP address in the vnet - however, az acr login throws this error - — Reply to this email directly, view it on GitHub <#17137 (comment)- 1127842531>, or unsubscribe <https://github.com/notifications/unsubscribe- auth/AVIA2WUDOTQPC4MBITOADXTVKJVGHANCNFSM4YIZXCPQ>. You are receiving this because you commented.Message ID: <Azure/azure- ***@***.***>

mdheerendra · 2022-05-20T11:38:23Z

I have this same issue, I've created a Azure Resourse Manager Service Connection and While using Azure CLI command az acr build --image reponame:imagename --registry acrname --file Dockerfile . with self hosted agent, I'm getting the same error.

ttasharski73 · 2022-05-23T18:38:08Z

I found that using docker login / docker build correctly uses the private endpoint.

I created a token on the ACR with a PW

Using this to login as example
echo $ACR_PW| docker login -u $ACR_USER --password-stdin registryUrl

This to build as example
docker build -t registryUrl/imagename --file DockerFile .

lordisp · 2022-07-06T22:26:06Z

I'm running private hosted devops agents. Login works but building the image using az acr build get still somehow routed over the public route:

az acr login --name myreg.azurecr.io
az acr build --registry myreg.azurecr.io --image xxx:yyy .

========================== Starting Command Output ===========================
/usr/bin/bash /agent/_work/_temp/771f368c-362a-4c11-8df9-b49ece019f58.sh
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
Login Succeeded
WARNING: The login server endpoint suffix '.azurecr.io' is automatically omitted.
WARNING: Packing source code into tar to upload...
WARNING: Excluding '.git' based on default ignore rules
WARNING: Excluding '.gitignore' based on default ignore rules
WARNING: Uploading archived source code from '/tmp/build_archive_c4dc880b67914c5d9e0fdabd3ca87293.tar.gz'...
WARNING: Sending context (854.722 KiB) to registry: myreg...
WARNING: Queued a build with ID: cb4
WARNING: Waiting for an agent...
2022/07/06 22:12:39 Downloading source code...
2022/07/06 22:12:40 Finished downloading source code
2022/07/06 22:12:41 Using acb_vol_055ad1ce-aff6-4195-bf76-f619d26a7abf as the home volume
2022/07/06 22:12:41 Setting up Docker configuration...
2022/07/06 22:12:41 Successfully set up Docker configuration
2022/07/06 22:12:41 Logging in to registry: myreg.azurecr.io

failed to login, ran out of retries: failed to set docker credentials: Error response from daemon: Get "[https://acrlhggixiservicesp.azurecr.io/v2/"](https://myreg.azurecr.io/v2/%22): denied: client with IP '20.50.200.13' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.
: exit status 1

The only way to get this working is building the image using docker build, tag it to the acr and push it:

az login --identity
az acr login --name myreg.azurecr.io
docker build -t foo:v1.0 .
docker tag foo:v1.0 myreg.azurecr.io/foo:v1.0
docker push myreg.azurecr.io/foo:v1.0

I believe this is a bug in az acr build cli

vf-doctor · 2022-08-29T01:49:43Z

Just want to add that I am also experiencing the same issue. Doing a nslookup or dig on the acr endpoint resolves to the private IP address but az acr build routes via the public endpoint.

And because I am running the agent in an azure container instance I have had problems getting docker to run inside the container. So the workaround noted above is not as easy for me to deploy.

thedheerendra · 2022-08-29T06:37:43Z

I had found with a Microsoft Case that you need to whitelist those 2,3 public IPs on ACR(with private endpoints) which will appear with that error. Those IPs are somehow useful for build.

vf-doctor · 2022-08-29T14:23:09Z

@thedheerendra Unfortunately, when you turn off public access entirely you no longer are able to whitelist any public IPs. The az acr build generates ACR tasks which use dedicated compute nodes to perform the container build. These virtual nodes exist outside the virtual network the build agent is running in. That seems to be the crux of the issue.

eiji102030 · 2022-11-15T11:33:03Z

I have the same issue. Are there any updates?

sbussetti · 2023-02-13T21:24:59Z

~~confirmed this issue on macos and ubuntu operating systems today~~
az acr login seems to work for me via private endpoint. It's az acr build that I have this issue with. It seems they have worked around this issue this way: Azure/acr#603

Wompipomp · 2023-02-15T09:46:57Z

We have the same issue. az acr build gets routed via public route :(

sbussetti · 2023-02-17T16:35:06Z

@Wompipomp see Azure/acr#603

flavian-anselmo · 2023-07-28T00:40:38Z

Any updates on this ?

toddysm · 2023-08-04T15:07:21Z

Changing the assignment to @northtyphoon and @terencet-dev for next steps.

rajeshkaremane · 2023-09-08T09:53:37Z

have same issue while running az cli on build agent and acr over private link

Work around
az acr login --name registryname
cd $(Build.SourcesDirectory)
docker buildx build -t registryname.azurecr.io/reponame:$(Build.BuildId) --platform=linux/arm64 .
docker push registryname.azurecr.io/reponame:$(Build.BuildId)
if ($LastExitCode -ne 0)
{
Write-Host “##vso[task.complete result=Failed;]DONE”
exit 1
}

AurimasNav · 2023-11-09T14:21:31Z

In my case it is az acr run --cmd "acr purge.." that is failing to work via private endpoint, probably the same root cause as for build.

Yeseh · 2023-11-14T13:46:33Z

have same issue while running az cli on build agent and acr over private link

Work around az acr login --name registryname cd $(Build.SourcesDirectory) docker buildx build -t registryname.azurecr.io/reponame:$(Build.BuildId) --platform=linux/arm64 . docker push registryname.azurecr.io/reponame:$(Build.BuildId) if ($LastExitCode -ne 0) { Write-Host “##vso[task.complete result=Failed;]DONE” exit 1 }

This problem seems to still be present. This workaround did the trick for us.

fortunkam · 2024-05-20T15:48:36Z

Did this ever get resolved? I am seeing the same exact problem as the original post. nslookup resolves the correct vnet address of the ACR, az acr login tries to work via the public address (and recommends adding my IP to the acr firewall)

ldvy · 2024-07-21T10:44:36Z

I have the same problem. The xxx.azurecr.io correctly resolves to xxx.privatelink.azurecr.io and goes through the internal endpoint to the ACR. However, upon running az acr login it seems to go over public internet.
Ridiculous that this hasn't been fixed for this long.

ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Feb 26, 2021

yungezz added Container Registry az acr Service Attention This issue is responsible by Azure service team. labels Mar 1, 2021

ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Mar 1, 2021

yugangw-msft self-assigned this Mar 25, 2021

yugangw-msft assigned toddysm and unassigned yugangw-msft May 16, 2021

gianlucagravina mentioned this issue Feb 21, 2022

Issue with ACR Build with private endpoint and private registry. Azure/acr#603

Closed

toddysm assigned northtyphoon and terencet-dev Aug 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

az acr login with private endpoint goes via public route #17137

az acr login with private endpoint goes via public route #17137

r3-jerrysteele commented Feb 26, 2021

ghost commented Mar 1, 2021

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

yungezz commented Mar 1, 2021

yugangw-msft commented Mar 25, 2021

fieldp commented Nov 18, 2021 •

edited

Loading

LahariChidura commented May 16, 2022

fieldp commented May 17, 2022 via email

mdheerendra commented May 20, 2022 •

edited

Loading

ttasharski73 commented May 23, 2022 •

edited

Loading

lordisp commented Jul 6, 2022 •

edited

Loading

vf-doctor commented Aug 29, 2022

thedheerendra commented Aug 29, 2022 •

edited

Loading

vf-doctor commented Aug 29, 2022 •

edited

Loading

eiji102030 commented Nov 15, 2022 •

edited

Loading

sbussetti commented Feb 13, 2023 •

edited

Loading

Wompipomp commented Feb 15, 2023

sbussetti commented Feb 17, 2023

flavian-anselmo commented Jul 28, 2023

toddysm commented Aug 4, 2023

rajeshkaremane commented Sep 8, 2023 •

edited

Loading

AurimasNav commented Nov 9, 2023

Yeseh commented Nov 14, 2023

fortunkam commented May 20, 2024

ldvy commented Jul 21, 2024

az acr login with private endpoint goes via public route #17137

az acr login with private endpoint goes via public route #17137

Comments

r3-jerrysteele commented Feb 26, 2021

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

ghost commented Mar 1, 2021

Describe the bug

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

yungezz commented Mar 1, 2021

yugangw-msft commented Mar 25, 2021

fieldp commented Nov 18, 2021 • edited Loading

LahariChidura commented May 16, 2022

fieldp commented May 17, 2022 via email

mdheerendra commented May 20, 2022 • edited Loading

ttasharski73 commented May 23, 2022 • edited Loading

lordisp commented Jul 6, 2022 • edited Loading

vf-doctor commented Aug 29, 2022

thedheerendra commented Aug 29, 2022 • edited Loading

vf-doctor commented Aug 29, 2022 • edited Loading

eiji102030 commented Nov 15, 2022 • edited Loading

sbussetti commented Feb 13, 2023 • edited Loading

Wompipomp commented Feb 15, 2023

sbussetti commented Feb 17, 2023

flavian-anselmo commented Jul 28, 2023

toddysm commented Aug 4, 2023

rajeshkaremane commented Sep 8, 2023 • edited Loading

AurimasNav commented Nov 9, 2023

Yeseh commented Nov 14, 2023

fortunkam commented May 20, 2024

ldvy commented Jul 21, 2024

fieldp commented Nov 18, 2021 •

edited

Loading

mdheerendra commented May 20, 2022 •

edited

Loading

ttasharski73 commented May 23, 2022 •

edited

Loading

lordisp commented Jul 6, 2022 •

edited

Loading

thedheerendra commented Aug 29, 2022 •

edited

Loading

vf-doctor commented Aug 29, 2022 •

edited

Loading

eiji102030 commented Nov 15, 2022 •

edited

Loading

sbussetti commented Feb 13, 2023 •

edited

Loading

rajeshkaremane commented Sep 8, 2023 •

edited

Loading