CRI v1 image API is not implemented for endpoint

[Cloud-Mak]

Hi All,

I have this k8s 1.23.8 cluster where CRI is containderd. I am trying to upgade it to 1.24.15 (latest in 1.24). I am getting below error while upgrading the 1st master (out of 3)

[preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.24.15: output: time="2023-06-18T23:18:53Z" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1

I have worked on containderd so far. Any idea what could be wrong here?

whenever I run crictl command I get below
root@stage-mast1:~# crictl ps -a

WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
E0619 05:03:17.124022 1995611 remote_runtime.go:390] "ListContainers with filter from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" filter="&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},}"
FATA[0000] listing containers: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 

I have tried to udpate containerd usign apt, but seems it only upadated client. I am not sure why it did not upadate server

root@stage-mast1:~# ctr version

Client:
Version: 1.6.12-0ubuntu1~20.04.1
Revision:
Go version: go1.18.1

Server:
Version: 1.5.9-0ubuntu1~20.04.6
Revision:
UUID: 580cee63-8afd-4437-9447-e44b0501167a
WARNING: version mismatch

Edit 1 - I have some other baremetal cluster which I upgraded to 1.24.15 from 1.23.12 with containerd 1.5.9

makrand@zmint:~$ kubectl get nodes -o wide
NAME         STATUS   ROLES           AGE    VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
demo-mast1   Ready    control-plane   250d   v1.24.15   10.20.4.98    <none>        Ubuntu 20.04.4 LTS   5.4.0-135-generic   containerd://1.5.9
demo-work1   Ready    <none>          250d   v1.24.15   10.20.4.102   <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.5.9

[AkihiroSuda]

Version: 1.5.9-0ubuntu1~20.04.6

For Kubernetes v1.24, you need containerd (Server) v1.6.4+ or 1.7.0+: https://github.com/containerd/containerd/blob/v1.7.2/RELEASES.md#kubernetes-support

The latest binaries can be downloaded from https://github.com/containerd/containerd/releases .
(Ubuntu 20.04 doesn't provide dpkg for v1.6 and newer. Docker provides dpkg for v1.6.x, but not for v1.7.x at this moment)

Also, if you have disabled_plugins = ["cri"] in /etc/containerd/config.toml, you have to remove it.

[Cloud-Mak]

Thansk for the answer.

config.toml only have disabled_plugins = []

BTW - I have some other baremetal cluster which I upgraded to 1.24.15 from 1.23.12 with containerd 1.5.9. This cluster was setup exact same as the cluster having upgrade issue. Upgraed cluster is at 1.23.9 and this one is at 1.23.12

makrand@zmint:~$ kubectl get nodes -o wide
NAME         STATUS   ROLES           AGE    VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
demo-mast1   Ready    control-plane   250d   v1.24.15   10.20.4.98    <none>        Ubuntu 20.04.4 LTS   5.4.0-135-generic   containerd://1.5.9
demo-work1   Ready    <none>          250d   v1.24.15   10.20.4.102   <none>        Ubuntu 20.04.4 LTS   5.4.0-113-generic   containerd://1.5.9

[superggn]

I'm using CentOS@7.9, but I believe that the directory structure and config are mostly the same between ubuntu and centos

1. crictl and ctr isn't using the same config.

1-1. crictl is using /etc/crictl.yaml, if this file doesn't present, you'll have to make one. here's an example

# these first two endpoint setting is where you configure crictl to containerd
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false

after creating crictl.yaml, there's no need to reload crictl like systemctl restart xxx, just use it directly is good

1-2. ctr is not recommended, and not using the containerd's config under /etc/containerd/config.toml, and I haven't figure out which config it should use

2. about the containerd config at /etc/containerd/config.toml

version = 2
# remember to delete this "disabled_plugins" variable, or set it to [], as the previous bro said
disabled_plugins = []
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "xxx"

for me because of my network environment, I'll have to set sandbox image separately.

after changing the containerd config, remember to systemctl restart containerd to load it

3. about the debug command
   systemctl status -l containerd this is my most frequently used command
   also journalctl -xeu kubelet provided by the kubeadm init error page

4. systemctl pre setup for containerd (make sure containerd is running)

systemctl enable containerd
systemctl start containerd
systemctl status containerd

Hope this could help

[Cloud-Mak]

Thanks. somehow missed to reply you back.

I ended up upgrading the containerd to 1.6.12. But the crictl command stopped working after that. Your /etc/crictl.yaml solved that.

[phantomsixthplayer]

@superggn @Cloud-Mak I had the following errors until I created /etc/crictl.yaml with:

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false

$ watch crictl info
time="2024-01-05T21:55:52+11:00" level=warning msg="runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead."
time="2024-01-05T21:55:52+11:00" level=error msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/dockershim.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory\""
time="2024-01-05T21:55:52+11:00" level=error msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/containerd/containerd.sock: connect: permission denied\""
time="2024-01-05T21:55:52+11:00" level=error msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///run/crio/crio.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /run/crio/crio.sock: connect: no such file or directory\""
time="2024-01-05T21:55:52+11:00" level=error msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/cri-dockerd.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory\""
time="2024-01-05T21:55:52+11:00" level=fatal msg="validate service connection: validate CRI v1 runtime API for endpoint \"unix:///var/run/cri-dockerd.sock\": rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory\""

I now only have this error, did you have this as well? If yes, what did you do to solve it?

$ crictl info
DEBU[0000] get runtime connection
FATA[0000] validate service connection: validate CRI v1 runtime API for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/containerd/containerd.sock: connect: permission denied"

$ sudo ls -l /var/run/containerd/
total 0
srw-rw----. 1 root root  0 Jan  5 22:15 containerd.sock
srw-rw----. 1 root root  0 Jan  5 22:15 containerd.sock.ttrpc
drwx--x--x. 2 root root 40 Jan  5 21:50 io.containerd.runtime.v1.linux
drwx--x--x. 2 root root 40 Jan  5 21:50 io.containerd.runtime.v2.task

$ sudo ls -l /run/containerd/
total 0
srw-rw----. 1 root root  0 Jan  5 22:15 containerd.sock
srw-rw----. 1 root root  0 Jan  5 22:15 containerd.sock.ttrpc
drwx--x--x. 2 root root 40 Jan  5 21:50 io.containerd.runtime.v1.linux
drwx--x--x. 2 root root 40 Jan  5 21:50 io.containerd.runtime.v2.task

Other configuration:

$ cat /etc/containerd/config.toml
disabled_plugins = []
root = "/var/lib/containerd"
state = "/var/run/containerd"

[grpc]
  address = "/var/run/containerd/containerd.sock"

Note: The default endpoints are now deprecated and the runtime endpoint should always be set instead. The performance maybe affected as each default connection attempt takes n-seconds to complete before timing out and going to the next in sequence.
https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

Runtime                             Path to Unix domain socket
containerd                          unix:///var/run/containerd/containerd.sock
CRI-O                               unix:///var/run/crio/crio.sock
DockerEngine (using cri-dockerd)    unix:///var/run/cri-dockerd.sock

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-runtime

[AkihiroSuda]

crictl needs sudo

[phantomsixthplayer]

Thanks mate, its as @AkihiroSuda said run sudo crictl info

Solution:

sudo tee /etc/crictl.yaml > /dev/null <<EOF
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false
EOF

I'm not sure if it matters but I also updated /etc/containerd/config.toml after generating the default config.

sudo sed -i 's|^state = ".*"|state = "/var/run/containerd"|' /etc/containerd/config.toml

sudo sed -i '/^\[grpc\]$/,/^\[/ s|^  address = ".*"|  address = "/var/run/containerd/containerd.sock"|' /etc/containerd/config.toml