Docker container failing to start on GKE

[isarkis]

I am facing a strange problem in GKE where OCI runtime creation is failing like so:

`

/usr/local/bin/docker create --name 353bb0779b0949a789aad2ce0c2fd4cf_alpine314_a47c67 --label 60e226 --workdir /__w/cobalt/cobalt --network github_network_09b19a3178cb4d4cb0e470134b93709d -e "HOME=/github/home" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/runner/_work":"/__w" -v "/runner/externals":"/__e":ro -v "/runner/_work/_temp":"/__w/_temp" -v "/runner/_work/_actions":"/__w/_actions" -v "/opt/hostedtoolcache":"/__t" -v "/runner/_work/_temp/_github_home":"/github/home" -v "/runner/_work/_temp/_github_workflow":"/github/workflow" --entrypoint "tail" alpine:3.14 "-f" "/dev/null"**

e4b70102da9157b4d03fee786560dd5dcf63294f02efc006d342631f982d9789

/usr/local/bin/docker start e4b70102da9157b4d03fee786560dd5dcf63294f02efc006d342631f982d9789**

Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/rdma/docker: permission denied: unknown
`

Oddly enough, this error only happens on the new nodes that were autoscaled recently. However, there are handful of nodes that were created last month which still work ok. So, all new nodes that get created experience this issue. I tried creating new cluster, but the problem persists.

What can be causing this permission error?

Node OS: Ubuntu 20.04.4 LTS with containerd
Node Kernel: 5.4.0-1065-gke
Kubelet Version: 1.21.10-gke.2000
Sysbox: 0.5.1 (also tried 0.4.1)

[rodnymolina]

@isarkis, thanks for filing this one up.

This is a dup of a recently-fixed issue (refer to #544 for details). We are about to publish a new release containing a fix for it (v0.5.2).

[rodnymolina]

Fixed in latest release. The sysbox-deploy-k8s images will be updated tomorrow.

@isarkis, please let us know if have any other questions. Closing issue now.

[isarkis]

@rodnymolina, thank you for getting back to me. I noticed 0.5.2 release has been published, but sysbox-deploy-k8s daemonset still points to 0.5.1.

[rodnymolina]

That's right @isarkis, i'm working on it (ETA: tomorrow morning if there are no surprises).

[rodnymolina]

@isarkis, sysbox-deploy-k8s images have been updated now. Please let us know if any issues.

[isarkis]

@rodnymolina, 0.5.2 is working as expected, thank you so much for your help!