containerd version not compatible with debian-10

[zetaab]

/kind bug

I am trying to update our clusters with 1.23 beta 1 kops to 1.23.1 kubernetes. However, debian 11 works fine but debian 10 does not work.

It seems that 1.23 beta 1 uses containerd 1.6.0-rc0, which is the issue. I went through the versions and it seems that 1.5.5 is latest containerd version that works with kOps in debian 10. All latest versions than that fails in debian 10

I cannot see any errors anywhere, the containers does not just start

root@master-espsti-obgs9w:/var/log# ctr -n k8s.io c ls
CONTAINER    IMAGE    RUNTIME
root@master-espsti-obgs9w:/var/log# ctr -n k8s.io i ls
REF                                                                                      TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                    LABELS
k8s.gcr.io/kops/dns-controller:1.23.0-beta.1                                             application/vnd.docker.distribution.manifest.v2+json      sha256:edb67ca94924170bf419e952b1527e4799d63ee7f4dab7b8f500e7e090ceacb5 112.0 MiB linux/amd64                                                                  io.cri-containerd.image=managed
k8s.gcr.io/kops/kops-controller:1.23.0-beta.1                                            application/vnd.docker.distribution.manifest.v2+json      sha256:a14881cd4b66189a19b1cfddba2820ee57f472e1245a67d3ebc7679cfae464a1 111.7 MiB linux/amd64                                                                  io.cri-containerd.image=managed
k8s.gcr.io/kops/kube-apiserver-healthcheck:1.23.0-beta.1                                 application/vnd.docker.distribution.manifest.v2+json      sha256:b129e809ea7b0d47b2d3d9a486078e4cab74ae061e8f8cfb20503bfe67cb05b3 26.6 MiB  linux/amd64                                                                  io.cri-containerd.image=managed
k8s.gcr.io/pause:3.6                                                                     application/vnd.docker.distribution.manifest.list.v2+json sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db 294.7 KiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,windows/amd64 io.cri-containerd.image=managed
k8s.gcr.io/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db application/vnd.docker.distribution.manifest.list.v2+json sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db 294.7 KiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,windows/amd64 io.cri-containerd.image=managed
sha256:3c3c400cc25b70b4c137d1455abb7dc9d22b9d0a4ceed40a8e93538747c6c594                  application/vnd.docker.distribution.manifest.v2+json      sha256:a14881cd4b66189a19b1cfddba2820ee57f472e1245a67d3ebc7679cfae464a1 111.7 MiB linux/amd64                                                                  io.cri-containerd.image=managed
sha256:4fff4a949aa15043548b62e3657c8b5c61320604b8932cac4b02adfeed3b35da                  application/vnd.docker.distribution.manifest.v2+json      sha256:edb67ca94924170bf419e952b1527e4799d63ee7f4dab7b8f500e7e090ceacb5 112.0 MiB linux/amd64                                                                  io.cri-containerd.image=managed
sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee                  application/vnd.docker.distribution.manifest.list.v2+json sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db 294.7 KiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,windows/amd64 io.cri-containerd.image=managed
sha256:d3a7e1baef5ca23ce50be2a6dea8498ebea0cf07ecd0ccff48279d86f27581fa                  application/vnd.docker.distribution.manifest.v2+json      sha256:b129e809ea7b0d47b2d3d9a486078e4cab74ae061e8f8cfb20503bfe67cb05b3 26.6 MiB  linux/amd64                                                                  io.cri-containerd.image=managed

this containerd/containerd#6447 is maybe related in 1.5.6 they updated libseccomp from 2.3.3 -> 2.5.1 which is maybe the issue here

[zetaab]

#13118 is maybe related to this

[hakman]

I think it's related to containerd/containerd#6209 (comment).