failed to create containerd task

[keyingliu]

Some of the containers cannot create with following error, the containerd we are running included PR #1072:

          starting container process caused "process_linux.go:424: container init
          caused \"rootfs_linux.go:58: mounting \\\"/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/549442c80aed91a09d82b7f8274d36290dca4a80993bf2ba4af0801344f8c8e3/hostname\\\"
          to rootfs \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7b6f62d1e8f31fc297777f45616c7e566275d40a1398ea3556bcd173d56a8806/rootfs\\\"
          at \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7b6f62d1e8f31fc297777f45616c7e566275d40a1398ea3556bcd173d56a8806/rootfs/etc/hostname\\\"
          caused \\\"not a directory\\\"\"": unknown

I can see the ${container_id}/hostname is a directory in the host, but it should be a file.

[keyingliu]

checked that the ${sandbox_id}/hostname is created during sandbox startup, but need to figure out why this file is missing...

[Random-Liu]

@keyingliu This shouldn't happen. We have an integration test testing this behavior https://github.com/containerd/cri/blob/master/integration/pod_hostname_test.go#L89.

What image are you using? It could be related to your image.

[keyingliu]

@Random-Liu I just picked one of our nodes, in /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes, run ls -R:

./1dfcec412f1752ca253b0208496e713861386e216798459ddd92265a1e170ab9:
hostname  hosts  resolv.conf

./358a9876da034f741649be1aa8262b65fd22bd1d67d63e77c77ec2c5039fe13d:
hosts  resolv.conf

./6c67c91e2a66c3d52ad80a001a32b642c7e8d121b98843028476e3e0f94e4c4b:
hostname  hosts  resolv.conf

./8224d8cba753739f6cc525154ac927b017b5a42d4cc22e9e498413928582ec8b:
hosts  resolv.conf

./90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd:
hostname  hosts  resolv.conf

./90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd/hostname:

./9182a2bf5d919bb520557b83f1963a4e552fb1933907bed281f3d230c2acbf79:
hosts  resolv.conf

./9407d3f3bc0099a77469e839dc044a06e0b2e8569d9def6513c10b1a884e1b6f:
hosts  resolv.conf

./a71b13e5fc3be7651a99e6ae688e094c12550073804699c9bbc4bb140b9dda9c:
hosts  resolv.conf

./aa1b8912f8a9052ccf0e5812d9f059f976c21d9cfea52cb9a37ca9bb8879405c:
hosts  resolv.conf

./ae42a6bf74489c8ad1643097827a0ec92ef9924b0adbd2a2ee9dffe9510e54f1:
hosts  resolv.conf

./af2b06fb2c3b18582ff8c730065bc12275311c2472c056b81936b4eb877ceef0:
hosts  resolv.conf

./ba55e2c222ceedcef812fd44ea685577912cfa32271792753a654997efb524f7:
hosts  resolv.conf

./cd54b904d55c36b6b1835fbd09a37cd5e8193d0ec6b0c7afc358c10c5cff9a4b:
hosts  resolv.conf

./d29cf2f181b0c3c836e78ddcd4b3bf4bb99a1c07a755ada4dabab01001f53b40:
hostname  hosts  resolv.conf

./d4cd2a46f8db6ee5c0a9bd545eba543be0d10dc3c95c3237a2bb214f94a648f4:
hostname  hosts  resolv.conf

./ee4239cbb7592d56344be7720a4ff4aabbe58be759f33742e3a74e42558c13b6:
hosts  resolv.conf

./f2052d65e620a9c5115571c4e8e97e4beb7f487d6ac5a02aef29f7d037df7f85:
hosts  resolv.conf

Take care the sandbox: 90b4739bb3f11aa74ada0b1c2adbf93c0ae5265f1ddbdf6c2a43ca391b2144bd, the hostname is a folder. and just wondering why some of the sandboxes have file hostname, but some of them missing the file.

[Random-Liu]

@keyingliu I see... Did you update containerd with all existing services running?

The hostname change is not a backward compatible change... If you have an old sandbox running, there won't be hostname file.

In this case, it is fine to not have hostname in container, but at least container should be able to start.

BTW, for production environment, it is always recommended to drain the node before updating critical node components. :)

[Random-Liu]

@keyingliu See #1085. Will include it in 1.2.5.

[keyingliu]

@Random-Liu Thanks for root cause analysis and fix.

[chengyli]

@Random-Liu When will containerd 1.2.5 be released?