Description

We've noticed that since the update from containerd v1.6.21 to v1.6.22 the systemd service did not start successfully. Debugging it closer reveled that containerd did not startup completely (missing "containerd successfully booted in ..." message) and did not send the sd notification READY=1 event.

In a quick test I've upgraded to v1.7.6, but the behavior remained the same.

The culprit turned out to be the CRI plug-in: After disabling it in containerd v1.6.22 and newer started up successfully.

The following error in the logs was pointing towards the CRI plug-in:

time="2023-09-24T21:43:20.958509027Z" level=warning msg="failed to load
plugin io.containerd.grpc.v1.cri" error="failed to create CRI service:
failed to create cni conf monitor for default: failed to create the
parent of the cni conf dir=/etc/cni: mkdir /etc/cni: read-only

We are using a read-only file system (squashfs), so the error seems reasonable. However, previous releases didn't had problems despite the CRI plug-in error.

Looking through the changelog of v1.6.22 makes this change the likely culprit: #8826

This is probably really an edge-case, but might still be worthwhile to fix.

Steps to reproduce the issue

1. Start containerd v1.6.22 or newer on a system with a read-only /etc
2. Observe the read-only error message on startup
3. Observe incomplete containerd start.

Describe the results you received and expected

Complete containerd start despite the read-only error.

What version of containerd are you using?

v1.6.22

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response