-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TLS error access denied when trying to pull images from cdn.mscr.io endpoint #19
Comments
@MRavenscroft Would you mind checking whether you are running behind a proxy? If so, can you compare the proxy configurations for a working and a non-working machine? |
I've checked the proxy settings and as far as i can tell i'm not behind one - my proxy options are turned off and i'm disconnected from our corporate VPN. Online proxy checks also cannot detect anything. |
@MRavenscroft apologies for the delay. Just wanted to check with you whether you are still experiencing the issue? |
No worries. Yep, we are still getting the issue same issue when we try to pull from there. We are working around it for now by building and hosting our own base images with the same contents as the official ones, but we'd much prefer to be able to use the official images. |
@MRavenscroft We are looking at this. Will keep you posted |
@MRavenscroft, Would you troubleshoot with curl or chrome browser to narrow down (or troubleshoot) the issue again? Here is how to do that. Each docker image layer can be downloaded with the following URL form. "https://mcr.microsoft.com/v2//blobs/" Supposing the repository path is "dotnet/core/sdk" and the layer's sha value is sha256:90fe46dd819953eb995f9cc9c326130abe9dd0b3993a998e12c01d0218a0b831, you can get the docker image layer with the below URL with any web browser such as Chrome. When you browse the URL, you will notice that it is redirected to a new URL which uses the "cdn.mscr.io" CDN end point that you have reported with the error message. Please notice that the redirected URL will contain the same sha value without "sha256:" prefix. If you still reproduce the issue with the same access denied error, please try the same url on other machines where the url works returning image data file successfully. And then, you might want to compare the two machine to check how the redirected URL pah is made. FYI, if the Azure region where the client is placed is different, the CDN endpoint will use a different one for each other. Ex. mcrneu0.cdn.mscr.io or mcrwcus0.cdn.mscr.io and etc. If you still can't figure out any hint or no difference between repro/non repro machines, please share the exact error message that you obtained from the brower with the url and the two URL addresses, one in the form of "https://mcr.microsoft.com/v2//blobs/" and the redirected URL. |
@MRavenscroft Can you confirm if the above URL matches the URL that you used when you received the error message? Anyway, I confirmed that I can download the image layer with "curl -L" as the below screenshot and confirmed the CDN endpoint is matched as well with "curl" without "-L".
|
Thanks, I've done some of that troubleshooting, The error i get when i try to pull for example the dotnet core SDK image is:
And from that, i built the url with the sha: https://mcr.microsoft.com/v2/dotnet/core/sdk/blobs/sha256:4aa6a74611ff353e9fd7ab05a3f837bfecb894592d3ae921bad52008def6fd2a Then, when i navigate or curl to that i get an SSL error saying i cannot connect to it When the member of the team who can connect to it tries either of those URLs (The original one with the sha, and the redirected one) and curling to it, it connects correctly for him, giving him a file to download and connection. Hope that helps. Not sure what the difference could be, as we have the same setup |
@MRavenscroft Did you check the certificate for the redirected URL? I sent the redirected URL from my machine. Even though I received "ERROR 403: Time-Limited URL validation", which is expected, I was able to see the certificate. It shows "*.cdn.mscr.io" for "Issued to:" as the below screenshot. Can you compare with this? If you received a different certificate for some reasons, that might be the reason of the SSL error. |
@MRavenscroft Would you execute the below two commands and send the result?
|
This is what i get from the nslookup (when not connected to the corporate VPN)
And this is what i get from the curl:
|
@MRavenscroft FYI, I am not sure but it seems that your machine has some issue on schannel module. According to the log you sent, it received only 7 byte out of 4096 byte and InitializeSecurityContext seems to be failed because it did not receive the required data for some reasons.
BTW, do you have any Linux (Ubuntu) shows the same problem? If so, would you execute the same command on Linux (such as Ubuntu) machine? Considering the fact that the Linux version curl can show more detailed information of the failure. |
Hi @MRavenscroft, do you have any update? |
Hi @jhkimnew , I had a look through the windows event log but couldnt find anything in there. I don't have a linux machine available, but did run a different installation of curl which has given a different result (I'm not sure whether the one i was running before came with Windows by default, or with my Git installation as i believe i read that git has curl built-in).
In case its useful, if i do the same curl on the redirected URL that is throwing the access denied when trying to do the docker pull, i get:
|
Hi @MRavenscroft, Didn't you use "-L" parameter when you execute curl? If you use "-L", the curl will follow redirects and you don't need to run it for the redirected URL. If you did not use the "-L" parameter, would you try again with "-L" and check if you still get the same access denied error? BTW, please give the full log and the command line you used so that I can understand what you tried and how to analyze the log.
Thanks, |
Hi @jhkimnew Sure thing, I've just double-checked, and the initial command I'd ran was the same one from above. The full log, including the command is:
|
Hi @MRavenscroft Honestly, I am not export on TLS issue. However, I think this is not a problem in MCR server side. MCR web server is mirrored in multiple regions. So, can you try the different region server instead of using your geographical region server? This is to confirm if or not you see the same issue with the different server. For example, in my region (westus), I can can the IP address of my region server with running "nslookup rpm0422wus.westus.cloudapp.azure.com". So, you can try use the westus instead of your region server.
mcr.microsoft.com
FYI, in case you curious how to find the specific host name (rpm0422wus.westus.cloudapp.azure.com), the answer is that I used "nslookup mcr.microsoft.com" to find the host name as the following screenshot shows.
|
@MRavenscroft I am closing this issue considering this issue seems to happen only from your machine or a specific networks issue. |
Hi,
We've been using the official microsoft images for our project that we are converting to use containers, and they were working a few months back. An example image that we are using is "mcr.microsoft.com/dotnet/core/sdk:3.1". However, some of our developers started running into a problem (I believe it was in Februrary initially) where we were getting access denied when trying to pull. when we try to pull the hello-world image from the "cdn.mscr.io" endpoint in https://github.com/microsoft/containerregistry/blob/master/client-firewall-rules.md, we get the same access denied error:
error pulling image configuration: Get https://mcrneu0.cdn.mscr.io/791e7ca5469f40b1b54c65b23e5dbde2-qgy0s4qedy//docker/registry/v2/blobs/sha256/09/095f049ec3a4c206f052648375b06599ce9d4332283bfd44ee99180c08df80f4/data?P1=1584975330&P2=1&P3=1&P4=jHPUhkNwRchm0jNoB4UjwQUNIAXMiZOtfVlvnec6bUs%3D&se=2020-03-23T14%3A55%3A30Z&sig=38Yi7vguA9F9jeX5kiuYxy4yIARQ50E0byn%2BakcA33w%3D&sp=r&sr=b&sv=2016-05-31®id=791e7ca5469f40b1b54c65b23e5dbde2: remote error: tls: access denied
When the endpoints for the images were moved to the ".data.mcr.microsoft.com" the pulling started to work for us, but now that the change was rolled back we are now getting the access denied error again.
Other things to note:
We've tried to identify whats causing it, but haven't had much success.
Is there anything else we could try (or a setup step we've missed?), or could there be an issue with the endpoint?
Thanks,
Mike
The text was updated successfully, but these errors were encountered: