hostPath does not work on containerd on Windows

[andyzhangx]

What happened:
hostPath does not work on containerd on Windows

I tried hostpath example with server:2004 on containerd on Windows and seeing this issue, it works well on docker Windows:

      Normal   Pulled   11s (x3 over 27s)  kubelet, 2088k8s00000001  Container image "eafdintregistry.azurecr.io/windows/servercore:2004" already present on machine
  Normal   Created  11s (x3 over 26s)  kubelet, 2088k8s00000001  Created container servercore-hostpath
  Warning  BackOff  10s (x3 over 23s)  kubelet, 2088k8s00000001  Back-off restarting failed container
  Warning  Failed   10s                kubelet, 2088k8s00000001  Error: failed to create containerd task: hcsshim::CreateComputeSystem servercore-hostpath: The parameter is incorrect.
(extra info: {​​​​​​​"Owner":"containerd-shim-runhcs-v1.exe","SchemaVersion":{​​​​​​​"Major":2,"Minor":1}​​​​​​​,"Container":{​​​​​​​"GuestOs":{​​​​​​​"HostName":"servercore-hostpath2"}​​​​​​​,"Storage":{​​​​​​​"Layers":[{​​​​​​​"Id":"8249eb58-a9b6-518c-a753-86b65e2f07f8","Path":"C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\335"}​​​​​​​,{​​​​​​​"Id":"ef78bbd9-5162-5a36-a6d8-30c0e4ad853a","Path":"C:\\ProgramData\\containerd\\root\\io.containerd.snapshotter.v1.windows\\snapshots\\334"}​​​​​​​],"Path":"\\\\?\\Volume{​​​​​​​88d5eee7-143e-4964-9aa7-afd9ef27520d}​​​​​​​\\"}​​​​​​​,"MappedDirectories":[{​​​​​​​"HostPath":"c:\\","ContainerPath":"d:."}​​​​​​​,{​​​​​​​"HostPath":"c:\\k","ContainerPath":"c:\\k"}​​​​​​​,{​​​​​​​"HostPath":"c:\\var","ContainerPath":"c:\\var"}​​​​​​​,{​​​​​​​"HostPath":"c:\\var\\lib\\kubelet\\pods\\033dfd99-1a52-4d07-8f30-bd3c5d90053e\\containers\\servercore-hostpath\\89a64e5f","ContainerPath":"c:\\dev\\termination-log"}​​​​​​​,{​​​​​​​"HostPath":"c:\\var\\lib\\kubelet\\pods\\033dfd99-1a52-4d07-8f30-bd3c5d90053e\\volumes\\kubernetes.io~secret\\default-token-llh9f","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true}​​​​​​​],"Networking":{​​​​​​​"Namespace":"1556dae8-4dec-4738-a806-d2d66d0c0c4e"}​​​​​​​}​​​​​​​,"ShouldTerminateOnLastHandleClosed":true}​​​​​​​): unknown

 Warning  Failed            5s         kubelet, k8s-master-40246862-0  Error: failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"cgroup\\\" to rootfs \\\"/run/containerd/io.containerd.runtime.v2.task/k8s.io/grafana/rootfs\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"stat /run/containerd/io.containerd.runtime.v2.task/k8s.io/44f86e838e827642911fc05e321384c5d93bbc614feda28967a3aa4738410b7d/rootfs/sys/besteffort/pod649f75a2-23ba-4af2-958c-376c04cd2545/grafana: no such file or directory\\\"\"": unknown

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/kind bug
/priority important-soon
/sig storage

[k8s-ci-robot]

@andyzhangx: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kevpar]

{​​​​​​​"HostPath":"c:\\","ContainerPath":"d:."}

I'm guessing this is the issue. Can you share what mounts you put in the pod spec?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[jsturtevant]

I am able to deploy a host path pod: https://raw.githubusercontent.com/jsturtevant/windows-k8s-playground/master/deployments/volumes/hostvolume.yaml

k get nodes -o wide   
NAME                    STATUS   ROLES    AGE   VERSION                             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
1213k8s000              Ready    agent    16m   v1.21.0-beta.1.156+def4aad28138cb   10.240.0.65    <none>        Windows Server 2019 Datacenter   10.0.17763.1637    containerd://1.4.3
1213k8s001              Ready    agent    16m   v1.21.0-beta.1.156+def4aad28138cb   10.240.0.4     <none>        Windows Server 2019 Datacenter   10.0.17763.1637    containerd://1.4.3


kubectl get pod                                                                                                                                                              
NAME      READY   STATUS    RESTARTS   AGE                                                                                                                                                                                                                                      
hostdir   1/1     Running   0          102s                   
                                                                                                                                                                                                                  
k exec -it hostdir cmd                                                                                                                                           
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.                                                                                                                                                
Microsoft Windows [Version 10.0.17763.1697]                                                                                                                                                                                                                                     
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>ls
License.txt     Program Files   ProgramData     Users           WcSandboxState  Windows         bin             curl            dev             dig             host            tmp             var

C:\>cd host

C:\host>ls
$RECYCLE.BIN               Documents and Settings     Packages                   Program Files              ProgramData                System Volume Information  Windows                    k                          release-notes.txt          var
AzureData                  NOTICE.txt                 PerfLogs                   Program Files (x86)        Recovery                   Users                      WindowsAzure               k.zip                      tmp                        vhd-id.txt

Going to close, if you can provide a reproduction please reopen.

/close

[k8s-ci-robot]

@jsturtevant: Closing this issue.

In response to this:

I am able to deploy a host path pod: https://raw.githubusercontent.com/jsturtevant/windows-k8s-playground/master/deployments/volumes/hostvolume.yaml

k get nodes -o wide   
NAME                    STATUS   ROLES    AGE   VERSION                             INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION     CONTAINER-RUNTIME
1213k8s000              Ready    agent    16m   v1.21.0-beta.1.156+def4aad28138cb   10.240.0.65    <none>        Windows Server 2019 Datacenter   10.0.17763.1637    containerd://1.4.3
1213k8s001              Ready    agent    16m   v1.21.0-beta.1.156+def4aad28138cb   10.240.0.4     <none>        Windows Server 2019 Datacenter   10.0.17763.1637    containerd://1.4.3


kubectl get pod                                                                                                                                                              
NAME      READY   STATUS    RESTARTS   AGE                                                                                                                                                                                                                                      
hostdir   1/1     Running   0          102s                   
                                                                                                                                                                                                                 
k exec -it hostdir cmd                                                                                                                                           
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.                                                                                                                                                
Microsoft Windows [Version 10.0.17763.1697]                                                                                                                                                                                                                                     
(c) 2018 Microsoft Corporation. All rights reserved.

C:\>ls
License.txt     Program Files   ProgramData     Users           WcSandboxState  Windows         bin             curl            dev             dig             host            tmp             var

C:\>cd host

C:\host>ls
$RECYCLE.BIN               Documents and Settings     Packages                   Program Files              ProgramData                System Volume Information  Windows                    k                          release-notes.txt          var
AzureData                  NOTICE.txt                 PerfLogs                   Program Files (x86)        Recovery                   Users                      WindowsAzure               k.zip                      tmp                        vhd-id.txt

Going to close, if you can provide a reproduction please reopen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andyzhangx]

now the following mountPath behavior changed from docker to containerd on Windows:

          volumeMounts:
          - mountPath: 'D:'
            name: azure

• docker on Windows

mount as a D drive

• containerd on Windows

mount as c:\D\ folder

question is how can I mount as a new drive with containerd on Windows? is this a regression?

[TBBle]

That does seem like a regression or bug. The mount path is supposed to be absolute, so I feel like containerd has probably done the wrong thing here.

If it really wanted to mount everything under C:\ (i.e. treating that as the root of system), then it should have either mounted at C:\D:\ (and almost certainly barfed due to invalid filename) or rejected it as an invalid filename earlier.

Probably needs to be a new bug report though, perhaps even against containerd, assuming the value came to it via CRI correctly.

[kevpar]

This is tracked via containerd/containerd#6589.