unable to start minikube with docker without cri-dockerd

[sudlo]

What Happened?

I have installed minikube on aws ec2 instance followed all the steps.
Please find the logs attached for your reference

Attach the log file

root@ip-172-31-47-92:~# cat logs.txt
*

• ==> Audit <==

• |---------|-------------------------|----------|------|---------|---------------------|---------------------|
  | Command | Args | Profile | User | Version | Start Time | End Time |
  |---------|-------------------------|----------|------|---------|---------------------|---------------------|
  | start | --vm-driver=none | minikube | root | v1.26.0 | 24 Jun 22 03:57 UTC | |
  | delete | | minikube | root | v1.26.0 | 24 Jun 22 04:00 UTC | 24 Jun 22 04:00 UTC |
  | start | --driver=docker | minikube | root | v1.26.0 | 24 Jun 22 04:00 UTC | |
  | delete | | minikube | root | v1.26.0 | 24 Jun 22 04:00 UTC | 24 Jun 22 04:00 UTC |
  | start | | minikube | root | v1.26.0 | 24 Jun 22 04:00 UTC | |
  | start | --force --driver=docker | minikube | root | v1.26.0 | 24 Jun 22 04:01 UTC | |
  | start | --driver=none | minikube | root | v1.26.0 | 24 Jun 22 04:01 UTC | |
  | start | | minikube | root | v1.26.0 | 24 Jun 22 04:02 UTC | |
  |---------|-------------------------|----------|------|---------|---------------------|---------------------|

• ==> Last Start <==

• Log file created at: 2022/06/24 04:02:49
  Running on machine: ip-172-31-47-92
  Binary: Built with gc go1.18.3 for linux/amd64
  Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
  I0624 04:02:49.109566 6897 out.go:296] Setting OutFile to fd 1 ...
  I0624 04:02:49.109700 6897 out.go:348] isatty.IsTerminal(1) = true
  I0624 04:02:49.109705 6897 out.go:309] Setting ErrFile to fd 2...
  I0624 04:02:49.109712 6897 out.go:348] isatty.IsTerminal(2) = true
  I0624 04:02:49.109921 6897 root.go:329] Updating PATH: /root/.minikube/bin
  W0624 04:02:49.110036 6897 root.go:307] Error reading config file at /root/.minikube/config/config.json: open /root/.minikube/config/config.json: no such file or directory
  I0624 04:02:49.110148 6897 out.go:303] Setting JSON to false
  I0624 04:02:49.110933 6897 start.go:115] hostinfo: {"hostname":"ip-172-31-47-92","uptime":493,"bootTime":1656042876,"procs":122,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"18.04","kernelVersion":"5.4.0-1078-aws","kernelArch":"x86_64","virtualizationSystem":"xen","virtualizationRole":"guest","hostId":"ec29327a-c75c-82bd-d5d2-09198152d1c4"}
  I0624 04:02:49.110994 6897 start.go:125] virtualization: xen guest
  I0624 04:02:49.113713 6897 out.go:177] 😄 minikube v1.26.0 on Ubuntu 18.04 (xen/amd64)
  W0624 04:02:49.116051 6897 preload.go:295] Failed to list preload files: open /root/.minikube/cache/preloaded-tarball: no such file or directory
  I0624 04:02:49.116256 6897 notify.go:193] Checking for updates...
  I0624 04:02:49.116293 6897 config.go:178] Loaded profile config "minikube": Driver=none, ContainerRuntime=docker, KubernetesVersion=v1.24.1
  I0624 04:02:49.116332 6897 driver.go:360] Setting default libvirt URI to qemu:///system
  I0624 04:02:49.116647 6897 exec_runner.go:51] Run: systemctl --version
  I0624 04:02:49.122833 6897 out.go:177] ✨ Using the none driver based on existing profile
  I0624 04:02:49.125045 6897 start.go:284] selected driver: none
  I0624 04:02:49.125059 6897 start.go:805] validating driver "none" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:none HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:resolv-conf Value:/run/systemd/resolve/resolv.conf}] ShouldLoadCachedImages:false EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name:m01 IP:172.31.47.92 Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/root:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
  I0624 04:02:49.125144 6897 start.go:816] status for none: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc: Version:}
  I0624 04:02:49.125166 6897 start.go:1541] auto setting extra-config to "kubelet.resolv-conf=/run/systemd/resolve/resolv.conf".
  I0624 04:02:49.125609 6897 cni.go:95] Creating CNI manager for ""
  I0624 04:02:49.125617 6897 cni.go:149] Driver none used, CNI unnecessary in this configuration, recommending no CNI
  I0624 04:02:49.125632 6897 start_flags.go:310] config:
  {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.32@sha256:9190bd2393eae887316c97a74370b7d5dad8f0b2ef91ac2662bc36f7ef8e0b95 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:none HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:resolv-conf Value:/run/systemd/resolve/resolv.conf}] ShouldLoadCachedImages:false EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name:m01 IP:172.31.47.92 Port:8443 KubernetesVersion:v1.24.1 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/root:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
  I0624 04:02:49.131832 6897 out.go:177] 👍 Starting control plane node minikube in cluster minikube
  I0624 04:02:49.134339 6897 profile.go:148] Saving config to /root/.minikube/profiles/minikube/config.json ...
  I0624 04:02:49.134543 6897 cache.go:208] Successfully downloaded all kic artifacts
  I0624 04:02:49.134579 6897 start.go:352] acquiring machines lock for minikube: {Name:mkc8ab01ad3ea83211c505c81a7ee49a8e3ecb89 Clock:{} Delay:500ms Timeout:13m0s Cancel:}
  I0624 04:02:49.134751 6897 start.go:356] acquired machines lock for "minikube" in 146.017µs
  I0624 04:02:49.134771 6897 start.go:94] Skipping create...Using existing machine configuration
  I0624 04:02:49.134778 6897 fix.go:55] fixHost starting: m01
  W0624 04:02:49.134971 6897 none.go:130] unable to get port: "minikube" does not appear in /root/.kube/config
  I0624 04:02:49.134981 6897 api_server.go:165] Checking apiserver status ...
  I0624 04:02:49.135007 6897 exec_runner.go:51] Run: sudo pgrep -xnf kube-apiserver.minikube.
  W0624 04:02:49.147114 6897 api_server.go:169] stopped: unable to get apiserver pid: sudo pgrep -xnf kube-apiserver.minikube.: exit status 1
  stdout:

stderr:
I0624 04:02:49.147151 6897 exec_runner.go:51] Run: sudo systemctl is-active --quiet service kubelet
I0624 04:02:49.156870 6897 fix.go:103] recreateIfNeeded on minikube: state=Stopped err=
W0624 04:02:49.156885 6897 fix.go:129] unexpected machine state, will restart:
I0624 04:02:49.159426 6897 out.go:177] 🔄 Restarting existing none bare metal machine for "minikube" ...
I0624 04:02:49.162619 6897 profile.go:148] Saving config to /root/.minikube/profiles/minikube/config.json ...
I0624 04:02:49.162753 6897 start.go:306] post-start starting for "minikube" (driver="none")
I0624 04:02:49.162762 6897 start.go:316] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0624 04:02:49.162803 6897 exec_runner.go:51] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0624 04:02:49.169850 6897 main.go:134] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0624 04:02:49.169872 6897 main.go:134] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0624 04:02:49.169883 6897 main.go:134] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0624 04:02:49.175670 6897 out.go:177] ℹ️ OS release is Ubuntu 18.04.6 LTS
I0624 04:02:49.180296 6897 filesync.go:126] Scanning /root/.minikube/addons for local assets ...
I0624 04:02:49.180353 6897 filesync.go:126] Scanning /root/.minikube/files for local assets ...
I0624 04:02:49.180373 6897 start.go:309] post-start completed in 17.610894ms
I0624 04:02:49.180382 6897 fix.go:57] fixHost completed within 45.605194ms
I0624 04:02:49.180388 6897 start.go:81] releasing machines lock for "minikube", held for 45.62773ms
I0624 04:02:49.180936 6897 exec_runner.go:51] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///var/run/cri-dockerd.sock
image-endpoint: unix:///var/run/cri-dockerd.sock
" | sudo tee /etc/crictl.yaml"
I0624 04:02:49.181000 6897 exec_runner.go:51] Run: curl -sS -m 2 https://k8s.gcr.io/
I0624 04:02:49.202779 6897 exec_runner.go:51] Run: sudo systemctl unmask docker.service
I0624 04:02:49.367152 6897 exec_runner.go:51] Run: sudo systemctl enable docker.socket
I0624 04:02:49.510093 6897 exec_runner.go:51] Run: sudo systemctl daemon-reload
I0624 04:02:49.686716 6897 exec_runner.go:51] Run: sudo systemctl restart docker
I0624 04:02:49.964699 6897 exec_runner.go:51] Run: sudo systemctl enable cri-docker.socket
I0624 04:02:49.977302 6897 out.go:177]
W0624 04:02:49.980229 6897 out.go:239] ❌ Exiting due to RUNTIME_ENABLE: sudo systemctl enable cri-docker.socket: exit status 1
stdout:

stderr:
Failed to enable unit: Unit file cri-docker.socket does not exist.

W0624 04:02:49.980260 6897 out.go:239]
W0624 04:02:49.984043 6897 out.go:239] ╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
I0624 04:02:49.986533 6897 out.go:177]

Operating System

Other

Driver

No response

[spowelljr]

Hi @sudlo, thanks for reporting your issue with minikube.

Kubernetes v1.24 dropped support for Dockershim, so if you want to use the combination of the none driver, Kubernetes v1.24+, and the Docker container runtime you'll need to install cri-dockerd on your system, as listed in our requirements page.

Other options include using a different driver, using an older version of Kubernetes --kubernetes-version, or using a different container runtime --container-runtime.

Please try one of those solution and let me know if it resolves your issue, thanks

[sudlo]

thanks a lot let me try it again

[afbjorklund]

Some more details in #14403, the docker driver should check for both cri-dockerd and crictl.

Currently it just looks for the "docker" client (?), but not for "dockerd" and "cri-dockerd" servers

// Available returns an error if it is not possible to use this runtime on a host
func (r *Docker) Available() error {
        _, err := exec.LookPath("docker")
        return err
}

[spowelljr]

Some more details in #14403, the docker driver should check for both cri-dockerd and crictl.

Currently it just looks for the "docker" client (?), but not for "dockerd" and "cri-dockerd" servers

// Available returns an error if it is not possible to use this runtime on a host
func (r *Docker) Available() error {
        _, err := exec.LookPath("docker")
        return err
}

That's a good point, I created an issue for this request #14460

[spowelljr]

So there are two error cases:

none driver

Users are unaware that minikube v1.26.0 requires a new dependency of cri-docker.

Background:

Kubernetes v1.24 dropped support for Dockershim, so if you want to use the combination of the none driver, Kubernetes v1.24+, and the Docker container runtime you'll need to install cri-dockerd on your system, as listed in our requirements page.

Other options include using a different driver, using an older version of Kubernetes --kubernetes-version, or using a different container runtime --container-runtime.

Implementing #14460 will detect that they're missing a system dependency. Will this prevent us from needing a solution message?

Solution:
Installing cri-docker, installation instructions: https://github.com/Mirantis/cri-dockerd#build-and-install

Other drivers (docker, etc.)

Users that try doing minikube start --kubernetes-version=stable (or v1.24.0+) on a cluster previously created before minikube v1.26.0 can result in failing to find cri-docker. I've tried reproducing it but oddly wasn't able to do so, but we have logs of users experiencing it (#14411).

Solution:
Run minikube delete then try starting minikube again and all should be resolved

[robrich]

Please link to this as a known issue in the 1.26.0 release page. Something like this:

Using a minikube cluster created before 1.26.0? After upgrading, you'll likely need to delete & recreate it. See issue 14410

[spowelljr]

Please link to this as a known issue in the 1.26.0 release page.

Updated, thanks for the idea

[shubham-cmyk]

You need to install the cri-dockerd,
We will build the cri-dockerd from scratch.

Clone the repo: git clone https://github.com/Mirantis/cri-dockerd.git

Install Golang Skip If present :

1. wget https://storage.googleapis.com/golang/getgo/installer_linux
2. chmod +x ./installer_linux
3. ./installer_linux
4. source ~/.bash_profile

Build the cri-dockerd :

1. cd cri-dockerd
2. mkdir bin
3. go get && go build -o bin/cri-dockerd
4. mkdir -p /usr/local/bin
5. install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
6. cp -a packaging/systemd/* /etc/systemd/system
7. sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
8. systemctl daemon-reload
9. systemctl enable cri-docker.service
10. systemctl enable --now cri-docker.socket`

[Technicalnikhi]

minikube
type: Control Plane
host: Stopped
kubelet: Stopped
apiserver: Stopped
kubeconfig: Stopped
getting error while checking status of minikube

[spowelljr]

Hi @Technicalnikhi, could you open a new issue and follow the template in the issue to upload all required logs, thanks

[shu-mutou]

cri-docker.service will always fails at first time.

Because sudo systemctl start cri-docker.service seems to create /etc/systemd/system/cri-docker.service.d/10-cni.conf, but the created file calls /usr/bin/cri-dockerd.

Despite the installation guide rewrite tha path to cri-dockerd. 😿

So we need to modify the path to the cri-dockerd in /etc/systemd/system/cri-docker.service.d/10-cni.conf after the first execution. Then run sudo systemctl daemon-reload && sudo systemctl restart cri-docker

[shu-mutou]

I don't know what timing, but 10-cni.conf seems to be restored. 😞
So in the installation guide, install ... should be set to /usr/bin/cri-dockerd and sed should not be executed.

[afbjorklund]

• cri-dockerd system unit override assumes /usr/bin/cri-dockerd but install instructions from mirantis point to /usr/local/bin/cri-dockerd #15265

[shu-mutou]

@afbjorklund Thank you so much for your explanation! I understand the current situation.

[owenallenaz]

Is it possible to get some clarification on whether cri-dockerd is required with using the --driver=docker. The docs don't mention it. If it truly is required it would be great if the official docs are updated. I'm receiving errors stating it's required using minikube 1.30.1 and kubectl 1.26.5.

[afbjorklund]

@owenallenaz : it is not required to install cri-dockerd on the host running the docker driver...
it is required in the image that runs kubernetes, but that one (kicbase) is provided by minikube