Unable to start k8 vm (`exec: "docker": executable file not found in $PATH`)

[mirza-ali-ctct]

Description

I am trying to bootstrap k8s VM on getting this issue:

[   89.492778] cloud-init[1361]: Cloud-init v. 21.4-0ubuntu1~20.04.1 finished at Mon, 11 Apr 2022 14:46:25 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 89.47 seconds
[FAILED] Failed to start Execute cloud user/final scripts.
See 'systemctl status cloud-final.service' for details.
[  OK  ] Reached target Cloud-init target.

[AkihiroSuda]

Any error in /var/log/cloud-init* in the guest ?

[mirza-ali-ctct]

Thats what I see in sudo cat /var/log/cloud-init-output.log

k8s.gcr.io/etcd:3.5.1-0
k8s.gcr.io/coredns/coredns:v1.8.6
+ kubeadm config images pull
docker is required for container runtime: exec: "docker": executable file not found in $PATH
To see the stack trace of this error execute with --v=5 or higher
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000001
LIMA| Exiting with code 1
Cloud-init v. 22.1-14-g2e17a0d6-0ubuntu1~20.04.2 running 'modules:final' at Tue, 12 Apr 2022 14:44:17 +0000. Up 17.55 seconds.
2022-04-12 14:45:48,303 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
2022-04-12 14:45:48,303 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
Cloud-init v. 22.1-14-g2e17a0d6-0ubuntu1~20.04.2 finished at Tue, 12 Apr 2022 14:45:48 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0][dsmode=net].  Up 107.97 seconds

[afbjorklund]

Hmm, that is weird... It should use crictl, not docker ?

[mirza-ali-ctct]

Hmm, that is weird... It should use crictl, not docker ?

I am using this example https://github.com/lima-vm/lima/blob/master/examples/k8s.yaml

[afbjorklund]

You can comment out these two lines, from the YAML:

    kubeadm config images list
    kubeadm config images pull

Then it should pull them using containerd CRI instead.

---

But it would still be nice know why it started failing...

I can't duplicate it here:

[   91.867127] cloud-init[1361]: + kubeadm config images list
[   92.508625] cloud-init[1361]: k8s.gcr.io/kube-apiserver:v1.23.5
[   92.509135] cloud-init[1361]: k8s.gcr.io/kube-controller-manager:v1.23.5
[   92.510601] cloud-init[1361]: k8s.gcr.io/kube-scheduler:v1.23.5
[   92.512588] cloud-init[1361]: k8s.gcr.io/kube-proxy:v1.23.5
[   92.515784] cloud-init[1361]: k8s.gcr.io/pause:3.6
[   92.517041] cloud-init[1361]: k8s.gcr.io/etcd:3.5.1-0
[   92.518567] cloud-init[1361]: k8s.gcr.io/coredns/coredns:v1.8.6
[   92.521023] cloud-init[1361]: + kubeadm config images pull
[   98.864912] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.23.5
[  OK  ] Stopped kubelet: The Kubernetes Node Agent.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  103.821153] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.23.5
[  107.127685] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/kube-scheduler:v1.23.5
[  OK  ] Stopped kubelet: The Kubernetes Node Agent.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  113.357558] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/kube-proxy:v1.23.5
[  115.233472] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/pause:3.6
[  OK  ] Stopped kubelet: The Kubernetes Node Agent.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  129.712686] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/etcd:3.5.1-0
[  OK  ] Stopped kubelet: The Kubernetes Node Agent.
[  OK  ] Started kubelet: The Kubernetes Node Agent.
[  137.968320] cloud-init[1361]: [config/images] Pulled k8s.gcr.io/coredns/coredns:v1.8.6
[  137.977848] cloud-init[1361]: + kubeadm init --cri-socket=/run/containerd/containerd.sock --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans 127.0.0.1

It should not be needed:

anders@lima-k8s:~$ kubeadm --v=5 config images pull
I0413 05:54:11.409525    5236 initconfiguration.go:117] detected and using CRI socket: /run/containerd/containerd.sock

But a workaround would be to make sure to pass --cri-socket to all three commands...

      --cri-socket string                    Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.

[afbjorklund]

What can happen is that it fails to detect the containerd socket, if containerd failed to start somehow.

Then Kubernetes v1.23.5 will still fall back to using the built-in dockershim, and look for "docker"

        switch len(foundCRISockets) {
        case 0:
                // Fall back to Docker if no CRI is detected, we can error out later on if we need it
                return constants.DefaultDockerCRISocket, nil
        case 1:
                // Precisely one CRI found, use that
                return foundCRISockets[0], nil
        default:
                // Multiple CRIs installed?
                return "", errors.Errorf("Found multiple CRI sockets, please use --cri-socket to select one: %s", strings.Join(foundCRISockets, ", "))
        }

        if criSocket != constants.DefaultDockerCRISocket {
                toolName = "crictl"
                // !!! temporary work around crictl warning:
                // Using "/var/run/crio/crio.sock" as endpoint is deprecated,
                // please consider using full url format "unix:///var/run/crio/crio.sock"
                if filepath.IsAbs(criSocket) && goruntime.GOOS != "windows" {
                        criSocket = "unix://" + criSocket
                }
                runtime = &CRIRuntime{execer, criSocket}
        } else {
                toolName = "docker"
                runtime = &DockerRuntime{execer}
        }

But then there should have been a lima error, when containerd failed to start ?

containerd:
  system: true
  user: false

[afbjorklund]

Need to update it anyway, the old config throws warnings in k8s 1.24

Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!

[afbjorklund]

@mirza-ali-ctct did you have any more logs ? Did the explicit cri-socket parameter work better ?

[afbjorklund]

This is fixed (--cri-socket)