Unable to launch Windows v1803 Image In Kubernetes on ACS

[sam-cogan]

Is this a request for help?:
No

Is this an ISSUE or FEATURE REQUEST? (choose one):
Issue

---

What version of acs-engine?:
0.15.1

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes 1.10

What happened:
When deploying an Windows Container using an 1803 image, on an 1803 node, it fails with the error : The operating system of the container does not match the operating system of the host. (0xc0370101)

Running the same container directly on the host machine using docker run works fine. I have confirmed both node and image are running 1803.

What you expected to happen:
Container to start

How to reproduce it (as minimally and precisely as possible):
Deploy an ACS Engine instance with Windows nodes based on the 1803 image, attempt to deploy a container using the microsoft/windowsservercore:1803 image.

Anything else we need to know:

Full error in Kubernetes;

Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "olfactory-olm-app-deployment-64ddfc96dd-f6z5w": Error response from daemon: container 7929bfd27250d5465496de2bb6a37c5e41d492488ca7c8362a07637b464b5e52 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {
    "SystemType": "Container",
    "Name": "7929bfd27250d5465496de2bb6a37c5e41d492488ca7c8362a07637b464b5e52",
    "Owner": "docker",
    "IsDummy": false,
    "VolumePath": "\\\\?\\Volume{0627cd58-20df-47a3-b494-8dd78e411995}",
    "IgnoreFlushesDuringBoot": true,
    "LayerFolderPath": "C:\\ProgramData\\docker\\windowsfilter\\7929bfd27250d5465496de2bb6a37c5e41d492488ca7c8362a07637b464b5e52",
    "Layers": [
        {
            "ID": "3a583df7-2f46-56a7-a452-7fc8abb7b534",
            "Path": "C:\\ProgramData\\docker\\windowsfilter\\89a92a6e8407219580d271d1689c7e479d26aa844ea92d60d9a0720eca6ab1a8"
        },
        {
            "ID": "2a988c67-9abf-5a34-85e1-0cfe5003064c",
            "Path": "C:\\ProgramData\\docker\\windowsfilter\\1a7a90531dbe455a4098d38b38793f0fd8b7f165ed8eae1f009492ef80aa3734"
        },
        {
            "ID": "6e5c7c2a-f3c3-5fd9-8040-bb2eeb5b6a9e",
            "Path": "C:\\ProgramData\\docker\\windowsfilter\\4ecd332ef1f0b0a519c8c1d640b42759919fe3d175e03f9e9ce490f482c87364"
        }
    ],
    "ProcessorWeight": 2,
    "HostName": "olfactory-olm-app-deployment-64ddfc96dd-f6z5w",
    "MappedDirectories": [],
    "HvPartition": false,
    "EnappintList": null,
    "Servicing": false,
    "AllowUnqualifiedDNSQuery": false
}

[jsturtevant]

Could you run ver on the host?

The error The operating system of the container does not match the operating system of the host. shows a miss match in some way. How did you deploy 1803 for the host? There was a bug (#2953) that didn't actually deploy 1803 even if you specified the 1803 image in the windowsProfile.

[sam-cogan]

Yeah, I saw that issue before I deployed so manually updated the ARM template to the correct version. Can confirm that 1803 is deployed on the host:

C:\Users\acsadmin>ver

Microsoft Windows [Version 10.0.17134.1]

The odd thing is, Docker its self is perfectly happy to deploy the same image on that host, manually doing that with docker run through the console works fine. It is just when I try to do it through Kubernetes.

Is Kubernetes getting the OS version from somewhere else other than directly from the host or anything?

[jsturtevant]

I just confirmed the same error. I ran docker images and found the kubletwin/pause image which I believe is used as part of container bootstrapping process. On inspection of that image it is 1709:

$ docker inspect kubletwin/pause

# output cropped for brevity
 "Architecture": "amd64",
  "Os": "windows",
  "OsVersion": "10.0.16299.431",

[jsturtevant]

Looks like

acs-engine/pkg/acsengine/defaults.go

Line 37 in 244d661

KubeBinariesSASURLBase: "https://acs-mirror.azureedge.net/wink8s/",

Contains the file for the kubletwin/pause Dockerfile which is built as part of the startup https://github.com/Azure/acs-engine/blob/master/parts/k8s/kuberneteswindowssetup.ps1#L216

I was able to change the kubletwin/pause Dockerfile to 1803 and deploy a 1803 images successfully through kubectl. I am not sure how the files are generated for the WindowsPackageSASURLBase. Anyone know where the code for that lives and how to update?

[jsturtevant]

@jackfrancis @JiangtianLi looks like the files for the windows binaries are not updated to support 1803. The pr #2958 won't work without updating the kubletwin/pause Dockerfile to 1803 but I can't find those files in this repo.

[jackfrancis]

@jsturtevant kubletwin/pause config is set in parts/k8s/kuberneteswindowssetup.ps1

[PatrickLang]

This Dockerfile should be just a few lines. I propose we inline it into https://github.com/Azure/acs-engine/blob/master/parts/k8s/kuberneteswindowssetup.ps1#L216 and set the right version

I don't know where it got put onto the acs-mirror, but here's the resulting file:

FROM microsoft/nanoserver:1709

CMD cmd /c ping -t localhost

[JiangtianLi]

@jsturtevant does 1803 image work on 1709 host?

[PatrickLang]

@JiangtianLi nope

I'm working on a proposed fix. Checking my version detection code right now

[JiangtianLi]

nvm, I saw all the places in acs-engine has changed to 1803, except the disabled test and documentation: https://github.com/Azure/acs-engine/search?utf8=%E2%9C%93&q=1709&type= We may change that too.

[jackfrancis]

Fixed by #2976 (thanks @PatrickLang!)