Cannot enforce NoNewPrivs: illegal version string v1

[stephan2012]

After switching Kubernetes v1.23.17 cluster nodes from dockershim to cri-dockerd, kubelet blocks a single Pod (the MetalLB controller) while all other Pods seem to run fine:

Mar 20 15:55:51 n0217 kubelet[1225]: E0320 15:55:51.965802    1225 pod_workers.go:965] "Error syncing pod, skipping" err="pod cannot be run: Cannot enforce NoNewPrivs: illegal version string \"v1\"" pod="metallb-system/controller-85dd57bdc-wh29s" podUID=578c94ac-0f44-40e2-bf74-9729c7806f18

Reverting to dockershim or setting securityContext.allowPrivilegeEscalation to true let the Pod start without issues. The Deployment does not look special:

        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - all
          readOnlyRootFilesystem: true

Using cri-dockerd also raises object not registered warnings.

  Warning  FailedMount  114s (x10 over 6m4s)  kubelet            MountVolume.SetUp failed for volume "cert" : object "metallb-system"/"webhook-server-cert" not registered
  Warning  FailedMount  114s (x10 over 6m4s)  kubelet            MountVolume.SetUp failed for volume "kube-api-access-pt487" : object "metallb-system"/"kube-root-ca.crt" not registered

However, they disappear after a while, and I wonder if they are related.

Is Kubernetes v1.23 compatible with cri-dockerd? Any hints on what’s missing or incompatible?

Versions:

• cri-dockerd v0.3.1
• OS: Ubuntu Server 22.04.2
• Linux kernel 5.19.0-35-generic
• Docker v23.0.1 (also tested with v20.10.23)

[afbjorklund]

There seems to be some hardcoded cases for DockerContainerRuntime, that are not applied to RemoteContainerRuntime. So the kubelet behaves differently, for dockershim and for cri-dockerd... As far as I know <= v1.23 still uses dockershim.

[stephan2012]

Indeed, updating the kubelet to v1.24 resolves the issues. So it looks like migrating to cri-dockerd before proceeding with the Kubernetes upgrade is not an option.

Can we add a note to the README that cri-dockerd is not (entirely) compatible with v1.23 for others entering the upgrade path?