failed to create containerd container: taking runtime copy of volume

[keyingliu]

Got error when creating pod:

failed to create containerd container: taking runtime copy of volume: open /var/lib/containerd/io.containerd.grpc.v1.cri/containers/1444a3e63095777cf0cd6ef348018ee0166d5e7999343c2fd01a8b6a2f92438b/volumes/458ce5eace3fe0ce7adfe2d639076f864fec3549af507d4c75d8c5dc623fb9cd: no such file or directory

roughly go through the code, in func (c *criService) generateVolumeMounts

		volumeID := util.GenerateID()
		src := filepath.Join(containerRootDir, "volumes", volumeID)

But I didn't see where the directory is created.

pod spec like:

spec:
  containers:
    image: myimage
    imagePullPolicy: Always
    name: test
    resources:
      limits:
        cpu: "2"
        memory: 8Gi
      requests:
        cpu: 1500m
        memory: 6Gi
    securityContext:
      privileged: false
      procMount: Default
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /home/jenkins/workspace
      name: volume-1
    - mountPath: /home/jenkins/.m2/hello2
      name: volume-0
    - mountPath: /home/jenkins
      name: workspace-volume
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qk2zh
      readOnly: true
    workingDir: /home/jenkins
  dnsPolicy: ClusterFirst
  nodeName: xxx
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - key: dedicated
    value: hello
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 900
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 900
  volumes:
  - hostPath:
      path: /tmp
      type: ""
    name: volume-0
  - emptyDir: {}
    name: volume-1
  - emptyDir: {}
    name: workspace-volume
  - name: default-token-qk2zh
    secret:
      defaultMode: 420
      secretName: default-token-qk2zh

[keyingliu]

the version of kubernetes is v1.12.3

[Random-Liu]

The host directory is created here: https://github.com/containerd/cri/blob/master/pkg/server/container_create.go#L688

And the error is generated here: https://github.com/containerd/cri/blob/master/pkg/containerd/opts/container.go#L100

Several questions:

1. Does this happen everytime you create the pod?
2. Can you provide your image? It is very important for reproducing this issue, because the issue happened to a image volume, which is specific to your image.

[keyingliu]

Actually the error is returned from:
https://github.com/containerd/cri/blob/master/pkg/containerd/opts/container.go#L112
The directory does not exist,the Readdir failed. Also in fs.copydir, the makedir is not enough, makedirall is needed. I will send the image later.

[Random-Liu]

@keyingliu We should have done the MkdirAll at https://github.com/containerd/cri/blob/master/pkg/server/container_create.go#L688.

We actually have an integration test for image volumes, and it works. https://github.com/containerd/cri/blob/master/integration/volume_copy_up_test.go

The image will be useful for troubleshooting this issue.

[keyingliu]

checked the dockerfile, there is only one VOLUME:
VOLUME /home/jenkins/workspace/
while in pod spec, there is volumemount:
/home/jenkins/workspace

And another different is, the /home and /tmp are soft link in host OS:
home -> var/home
tmp -> sysroot/tmp

[keyingliu]

guess it might be here: https://github.com/containerd/cri/blob/master/pkg/server/helpers.go#L377
it is not same as https://github.com/containerd/cri/blob/master/pkg/server/container_create.go#L1068

[Random-Liu]

@keyingliu Thanks! I think it is because of the CleanPath, we use it when merging CRI mounts and extra mounts, but not when generating the volume mounts.

You have /home/jenkins/workspace/ in your image, but /home/jenkins/workspace in your pod spec, so it is excluded when generating host directory, but included when copy volume content, thus this error.

Can you use /home/jenkins/workspace/ in your pod spec for now? Or /home/jenkins/workspace in your image?

I will send out a fix and cherry-pick to all supported versions, and rollout a patch release next week. Thanks for the bug report!

[keyingliu]

@Random-Liu sure, we will use the workaround you mentioned, thanks!