cri-tools 1.26 incompatible with containerd 1.5 #1130
Comments
yogeek
added
kind/bug
|
You need to pin crictl as well, once you figure out what version to use Like |
|
@afbjorklund crictl is not included in cri-tools package...? And regarding my initial question, do you confirm that cri-tools not being compatible with containers 1.5 is an bug ? |
|
Sorry, my bad. Sometimes the package for
It is not a bug, support for CRI v1alpha2 was deliberately removed in Kubernetes 1.26 https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed The "bug"/missing feature, is pinning the kubeadm requirements on cri-tools to match
|
|
Trying to come up with a good way to match them, right now release repo is the best "guess" We probably want to pre-compute a JSON file (or such) for caching purposes, like with the others
|
|
The old version of cri-tools is having similar issues communicating with newer runtimes, such as CRI-O. Fortunately the default cri-o version lags a bit, so it still "works" when using 1.24-1.25 instead of 1.26. I guess we need to stop using packages for it, or at least change the repository away from Kubic. docker@minikube:~$ apt list cri-tools
Listing... Done
cri-tools/unknown,now 1.21.0~2 amd64 [installed]
cri-tools/unknown 1.21.0~2 arm64
cri-tools/unknown 1.21.0~2 armhf
cri-tools/unknown 1.21.0~2 s390x
docker@minikube:~$ apt list cri-o
Listing... Done
cri-o/unknown,now 1.25.3~0 amd64 [installed]
cri-o/unknown 1.25.3~0 arm64
cri-o/unknown 1.25.3~0 armhf
cri-o/unknown 1.25.3~0 s390x
docker@minikube:~$ apt-cache madison cri-tools
cri-tools | 1.21.0~2 | http://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 Packages
docker@minikube:~$ apt-cache madison cri-o
cri-o | 1.25.3~0 | https://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.25/xUbuntu_20.04 PackagesBoth docker and containerd still support alpha2, but I guess they too will drop support soon (v2.0). |
|
Same issue as: |
|
@afbjorklund thanks for the link to the revert issue. But it seems that this was closed whereas the discussion was not finished... Or do I misunderstood something ? |
|
Yes, I don't think the underlying issue was resolved... It is up to the Kubernetes user to set up and configure CRI and CNI, including finding a version that works and changing required configuration such as socket paths or cgroup version. In your case, I think it needs pinning the version (like the kube debs) ? |
|
Ok...it is a pity that the underlying dependency issue in kubeadm was not fixed but apparently it does not seem so easy (I do not understand all the implications to be honest) For now, and for those who will have the same problem, here is the workaround we did to make it work when installing a 1.22 K8s cluster :
Thank you for your help ! |
|
The versions are a bit hard to guess, Kubernetes 1.27.0 uses cri-tools 1.26.0 and cni-plugins 1.2.0 Easy now (latest/greatest), but it might be harder to align in the future when it (v1.27) too is old |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
|
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/reopen |
|
@yogeek: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
removed
the
lifecycle/rotten
|
Seems unlikely that this will get done, now that the old packages are removed: https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/ The new packages have one repository per release, so uses different CRI and CNI https://build.opensuse.org/project/subprojects/isv:kubernetes |
|
Ok @afbjorklund thanks for the update 👍 |
|
I spoke too soon, the old pinning was just copied over to the new repos. So v1.25 still installs cri-tools 1.26... But for the newer releases (1.28 and 1.29), they were supposed to be forked for each of the repositories. cri-tools/unknown 1.28.0-1.1 amd64 cri-tools/unknown 1.29.0-1.1 amd64 |
What happened:
We deploy K8S clusters on AWS EC2 instances with kubeadm
We are trying to follow an immutable pattern by :
Recently, we built a new base image to benefit from the latest CSI hardening without changing anything in our scripts (still installing the same pinned packaged versions, only the base ubuntu changed) and we were surprised to have a broken cluster as a result, resulting from error in kubeadm command.
After some investigations, we found out that "cri-tools" package (that we do not install explicitely) was implicitely installed as a kubeadm dependency, but apparently, even without modifying the kubeadm version, the installation of kubeadm in our new AMI installed a newer version of "cri-tools"... before we had cri-tools 1.25 package and we ended with cri-tools 1.26.
And cri-tools 1.26 is apparently not compatible with containerd 1.5... which seem to be the root cause of our kubeadm isssue
As soon as cri-tools 1.26 was released, any installation of kubeadm would pull it as a dependency and cause the issue with containerd.
To solve it, we thought about 2 solutions :
What you expected to happen:
I would expect :
Anything else we need to know?:
I was not sure if I should open the issue here or in the kubeadm repository...
Environment:
containerd 1.5.11-1cat /etc/os-release):uname -a):The text was updated successfully, but these errors were encountered: