"Access is denied" when pulling Windows Nano Server container images (microsoft/aspnetcore:2.0-nanoserver-1709)

[DaveAurionix]

When performing a Kubernetes Deployment of a Windows Nano Server and .NET Core 2.0 container onto a Windows node managed by an AKS Kubernetes cluster, the following error is consistently thrown:

Failed to pull image "microsoft/aspnetcore:2.0-nanoserver-1709": rpc error: code = Unknown desc = failed to register layer: re-exec error: exit status 1: output: remove \?\C:\ProgramData\docker\windowsfilter<large hex string>\UtilityVM\Files\Windows\System32\diagtrack.dll: Access is denied.

The same image pull works outside of AKS and produces usable containers, e.g. locally using Docker for Windows.

I was wondering if there was some security configuration on each node that was required and that I was missing in the ARM template JSON used to create the AKS managed cluster.

[DaveAurionix]

Extremely misleading error. Seems related to the amount of RAM configured against the container in the requests block of the Kubernetes YAML. Too little RAM (i.e. default amount) will result in this error. Increasing the RAM results in the container being created correctly.

[DaveAurionix]

Apologies, actually this is an intermittent error that occurs often. Increasing available RAM may reduce the frequency a little. It seems that the microsoft/dotnet:2.0.3-runtime-nanoserver-1709 image will almost always cause pods to fail with ErrImagePull (detail in opening post). However the microsoft/nanoserver:1709 image that it is directly based on will almost always succeed, despite very few (no relevant?) differences between those images.

Due to the intermittent nature, I suspect that AKS is selecting a virtual machine image for Windows agent VMs that has anti-malware installed and that results in any Windows containers running a risk of Access is denied errors whilst image layers are being pulled and applied (i.e. before the container is able to start). Possibly the mechanism that is pulling and applying images needs to be tolerant of this and retry appropriately, or the base agent VM image used by Azure Container Services when spinning up nodes needs adjusting.

Strangely, once a pod fails pulling an image then that same pod will fail on every retry (seemingly).

[bremnes]

FYI: Regarding the "diagtrack.dll: Access is denied", we have a similar error over at ACS: #88. This happens for clusters created through the portal and using ARM template. (Just created a Microsoft support ticket for it, will notify if we figure something out.)

[sioakim]

I'm interested to run AKS with Windows containers as well. But I didn't see if there is support to create AKS with Windows. I saw integration with ACI for Windows containers.

Can you please let me know your setup? Is it a separate unmanaged ACS installation with Windows support that you managed trough the AKS installation?

[DaveAurionix]

@sioakim It's a managed AKS cluster containing some Linux and some Windows nodes. I admit I can't quickly find the URLs to the articles/blogs I was copying this from, but the basic principle is as follows:

1. In the ARM JSON for the AKS cluster there is a linuxProfile section, but you can carefully copy windowsProfile examples over from ACS (plus any related JSON) into AKS and make some progress.
2. There is an article about hosting IIS in a Windows container on ACS with unmanaged Kubernetes which I think I based most of the JSON on and copied it into an AKS ARM template.
3. AKS seems to "just" add a management layer on top of ACS - at the moment, whilst in preview, you can even see the "behind-the-scenes" resource group created (with an MC_ prefix) which appears to be heavily ACS based so I would expect a high degree of similarity in the ARM JSON for the two.
4. The actual Kubernetes documentation helps a lot with Windows Container support on Kubernetes too when trying to figure the configuration out for reproduction in the ARM JSON.

[slack]

Registering a windows kubelet against an AKS cluster will work, but isn't something we support.

Real windows support is on our roadmap for 2018. Closing this issue.

[tugberkugurlu]

@slack are Windows Containers supported with AKS today?