-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Image pull errors: failed to do request: Head "http://localhost:8578/v2/..." when streaming not required #739
Comments
Thanks for reporting this. Does the nodepool have access to both registries? |
Hi, yes we have a service principal tied to the AKS cluster that has pull permissions for both ACRs |
Got it, and can you confirm a couple of things to help narrow down the root cause.
Meanwhile, I will try to reproduce the issue on my end as well. |
I can confirm there are no issues if artifact streaming is disabled in the cluster, we have only started seeing this since we enabled it and we only see it for the one nodepool that we enabled it on. I was able to get the image to pull from the same ACR when it was not using streaming. However the issue appears to be temporamental and hard to reproduce as it affects random nodes (even though they have not interacted with our streaming-enabled ACR at any point) Could you please share the command to disable artifact streaming on a nodepool, I could disable it on the pool we are seeing issues and validate that the issue goes away |
@awalford16 I'm able to reproduce this on my side, so I'll be working on a fix. ETA, for a fix will be some time in the next month. |
@awalford16 so I did a bit more digging and I have a workaround you could try. It appears to happen when you have the same image-reference w/ different registries in the same pod. For example, Fails apiVersion: v1
kind: Pod
metadata:
name: &name mix-wordpress
spec:
containers:
- name: wordpress-streaming
image: streaming.azurecr.io/wordpress:latest
- name: wordpress-nonstreaming
image: non-streaming.azurecr.io/wordpress:latest Works apiVersion: v1
kind: Pod
metadata:
name: non-wordpress
spec:
containers:
- name: wordpress-nonstreaming
image: non-streaming.azurecr.io/wordpress:latest Works apiVersion: v1
kind: Pod
metadata:
name: wordpress
spec:
containers:
- name: wordpress-streaming
image: streaming.azurecr.io/wordpress:latest (I tested these all running on the same node pool w/ node-selectors) I am working on figuring out the root cause and a fix, but just wanted to share a possible workaround you could try for your own evaluation. |
So it looks like this can affect any pod spec that has multiple containers. It looks like there's been a regression in AKS/containerd but I'm still trying to narrow it down. |
@awalford16 Could you provide the value of this label from your nodepool that has this issue, `kubernetes.azure.com/node-image-version' For example it should be some value that looks like this - |
@juliusl thanks for looking into this. The label value is |
@awalford16 so good news, I figured out the issue and I have a fix. I'm working on the release, so should be about a week or two for it to make it's way upstream. |
@awalford16 Hey there just to close up the loop. The fix has been rolled out to all AKS regions for about a week or two, are you able to update your node images and give it a try? |
Thanks @juliusl! Looks like it is working on my end now. For confirmation these are the versions on my nodes: |
Has the fix (#739) been distributed to UK South? And how can I rollback "az aks nodepool update --enable-artifact-streaming"? |
@awalford16 thanks for following up! |
@tolga-hmcts I'm getting someone from the AKS side to chime in on that
Yes, it should have been rolled out at the time of your comment |
Describe the bug
We have been experimenting with ACR streaming feature and while doing this we set up a new ACR, copied some images across and enabled streaming on it.
We then enabled a nodepool to use acr streaming and since then we have seen some pod deployments fail with:
The pod deployments that are failing are trying to pull from a different ACR (without streaming enabled) but they are using the same nodepool that we were using to test the streaming. I have tried scaling down the nodes and deploying streaming pods onto a different nodepool but we are still seeing this error with odd nodes that never had a streaming pod there.
When this happens the pod never successfully pulls, it continuously fails and we have to delete the node from the cluster
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
All pod deployments, whether coming from Artifact Streaming or standard image pull should work as normal
Screenshots
If applicable, add screenshots to help explain your problem.
Any relevant environment information
Additional context
Add any other context about the problem here.
If any information is a concern to post here, you can create a support ticket or send an email to acrsup@microsoft.com.
The text was updated successfully, but these errors were encountered: