Closed
Description
On our sysbox enabled gitlab runners we encounter the following warning and errors after some time and from then on no new containers can be started.
This warning in sysbox-fs appears shortly before containers cannot be started anymore:
Apr 20 14:16:03 mam-sysbox-bs01 sysbox-fs[840]: time="2021-04-20 14:16:03" level=warning msg="Sysbox-fs first child process error status: pid = 30240"
Apr 20 14:16:03 mam-sysbox-bs01 sysbox-fs[840]: time="2021-04-20 14:16:03" level=warning msg="Error during syscall \"mount\" processing on fd 10 pid 29943 (mountInfoParser error for pid = 29943: Error waiting for sysbox-fs first child process)"
These errors appear shortly afterwards, here i grepped for the container id:
# journalctl -u sysbox-mgr.service -a | grep 4eccbcdcu
Apr 20 14:16:42 mam-sysbox-bs01 sysbox-mgr[641]: time="2021-04-20 14:16:42" level=info msg="registered new container 4eccbcdc964c"
Apr 20 14:16:53 mam-sysbox-bs01 sysbox-mgr[641]: time="2021-04-20 14:16:53" level=info msg="unregistered container 4eccbcdc964c"
Apr 20 14:16:53 mam-sysbox-bs01 sysbox-mgr[641]: time="2021-04-20 14:16:53" level=info msg="released resources for container 4eccbcdc964c"
# journalctl -u sysbox-fs.service -a | grep 4eccbcdc
# journalctl -u docker.service -a | grep 4eccbcdc
Apr 20 14:16:42 mam-sysbox-bs01 dockerd[1103]: time="2021-04-20T14:16:42.828288250Z" level=warning msg="error registering link for 4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06, to 34a314781baafd3022ece12a26c033ebadb6137c448d5bcf057f5ba08e067d7e, as alias docker.mamdev.server.lan-internal-mamido-docker-dind, ignoring: name is reserved"
Apr 20 14:16:53 mam-sysbox-bs01 dockerd[1103]: time="2021-04-20T14:16:53.554308212Z" level=error msg="4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06 cleanup: failed to delete container from containerd: no such container"
Apr 20 14:16:53 mam-sysbox-bs01 dockerd[1103]: time="2021-04-20T14:16:53.554373829Z" level=error msg="Handler for POST /v1.25/containers/4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06/start returned error: OCI runtime create failed: container_linux.go:392: starting container process caused: process_linux.go:592: container init caused: process_linux.go:648: registering with sysbox-fs caused: failed to register with sysbox-fs: failed to register container with sysbox-fs: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown"
Apr 20 14:16:53 mam-sysbox-bs01 dockerd[1103]: time="2021-04-20T14:16:53.554861317Z" level=error msg="Handler for GET /v1.41/containers/4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
# journalctl -u containerd.service -a | grep 4eccbcdc
Apr 20 14:16:42 mam-sysbox-bs01 containerd[971]: time="2021-04-20T14:16:42.867436461Z" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06 pid=30846
Apr 20 14:16:53 mam-sysbox-bs01 containerd[971]: time="2021-04-20T14:16:53.401887443Z" level=info msg="shim disconnected" id=4eccbcdc964cbfa38f577dede0595fdb1b11bd9e9756db81643897ac55bfde06
The error message regarding DeadlineExceeded is the same as in these issues:
#244
#176
#72
This time there is no mount like in #244 involved.
systemctl status shows sysbox, sysbox-fs, sysbox-mgr, docker, containerd working but every start of a container fails with the DeadlineExceeded message.