Cannot exec into container running in gVisor

[ianlewis]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Please provide the following details:

Environment:

Minikube version (use minikube version): 0.31.0

• OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS (Xenial Xerus)
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): kvm2
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): 0.31.0

What happened:
Run nginx in gvisor per the instructions for the gVisor addon: https://github.com/kubernetes/minikube/tree/master/deploy/addons/gvisor

$ kubectl exec -ti nginx-untrusted -- /bin/sh                                                                                                                 
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "c875f37690783db80623f10cddd6c51feda8a8fa9216be73edb44f700f61d18f": failed to retrieve OCI runtime exec internal pid: open /run/containerd/io.containerd.runtime.v1.linux/k8s.io/8f65ab7b4275beb681570544ab112969eb0a02f1cf56ea6a40552f179025df0f/c875f37690783db80623f10cddd6c51feda8a8fa9216be73edb44f700f61d18f-internal.pid: no such file or directory: unknown

What you expected to happen:
I can exec a terminal in the container

How to reproduce it (as minimally and precisely as possible):

minikube start
minikube addons enable gvisor
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  containers:
  - name: nginx
    image: nginx
EOF
kubectl exec -ti nginx-untrusted -- /bin/sh

[ianlewis]

This doesn't seem consistent. If I run kubectl exec into a normal trusted container and then exec into the untrusted gvisor container then it works.

i.e. after reproducing per above

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-trusted
spec:
  containers:
  - name: nginx
    image: nginx
EOF
pod/nginx-trusted created

$ kubectl exec -ti nginx-trusted -- /bin/sh
# exit

$ kubectl exec -ti nginx-untrusted -- /bin/sh
# exit

[balopat]

@ianlewis yes, I saw this too (google/gvisor#120) - not sure which layer is throwing the error - but it seems that is related to the gvisor-containerd-shim or runsc, as it does not come up with trusted workloads. Still investigating.

[ianlewis]

Thanks, It's very possible that it's the shim but I hadn't encountered it before in my testing on my own Kubernetes clusters. I'll also investigate a bit today to see if I can't find the root cause.

[ianlewis]

@balopat see my other comment too but let's keep comments on the issue here for now since minikube is currently how we can repro and move to the shim or gvisor once we have a cause.

[ianlewis]

This looks to be a race condition in the gvisor-containerd-shim where runsc hasn't yet created the pid file for the process executed by 'kubectl etc' but gvisor-containerd-shim tries to read it. Moving to the shim repo.

[ianlewis]

Actually, I'll keep this open as it's proving a bit challenging to repro using strait containerd and crictl, will continue to try to reproduce outside of minikube though.

[ianlewis]

We will also need to update minikube with a newer version of the shim and/or containerd so we can track that work on this issue.

[ianlewis]

@balopat You can maybe just assign this issue to me. I'll provide the proper PR to update minikube once the shim is updated.

[ianlewis]

@balopat This is a minikube issue. I investigated on google/gvisor-containerd-shim#4 and found that this is fixed in runsc by google/gvisor@4cd4b60 but minikube is using an old version of runsc.

The version of runsc used by the gvisor addon's docker image doesn't match the version used by the current code at https://github.com/kubernetes/minikube/blob/master/pkg/minikube/constants/constants.go#L285. The code here is referencing the 2018-12-04 version of runsc but the gvisor addon binary in the docker image is using 2018-11-08.

$ for m in $(seq -f "%02g" 1 12); do
>     for d in $(seq -f "%02g" 1 31); do
>         if grep "https://storage.googleapis.com/gvisor/releases/nightly/2018-${m}-${d}/runsc" gvisor-addon; then
>             echo "2018-${m}-${d}";
>         fi;
>     done;
> done;

Binary file gvisor-addon matches
2018-11-08

Please rebuild and deploy the gcr.io/k8s-minikube/gvisor-addon docker image to use at least the 2018-12-04 version of gVisor.

[balopat]

Thank you for the investigation! I pushed a new version of the image built from master and it works now. Closing.