Image garbage collection in K8S?

[nyxi]

So with Kubernetes you are supposed to let kubelet handle garbage collection of images.

But I can't seem to figure out how to configure that with cri-o as the container runtime? For my nodes I just get a lot of:

Warning ImageGCFailed 2s (x1416 over 4d21h) kubelet failed to get imageFs info: non-existent label "crio-images"

For now I had to implement a simple systemd service/timer to run crictl rmi --prune.

Kubernetes 1.19.3
cri-o 1.19.0

[haircommander]

what args do you use to run the kubelet, what's the output of crio config and crio-status info ?

[nyxi]

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
--kubeconfig=/etc/kubernetes/kubelet.conf
--config=/var/lib/kubelet/config.yaml
--container-runtime=remote
--container-runtime-endpoint=unix:///var/run/crio/crio.sock
--dynamic-config-dir=/var/lib/kubelet/dynamic-config

/var/lib/kubelet/config.yaml

crio config

$ crio-status info
cgroup driver: systemd
storage driver: overlay
storage root: /var/lib/containers/storage
default GID mappings (format <container>:<host>:<size>):
  0:0:4294967295
default UID mappings (format <container>:<host>:<size>):
  0:0:4294967295

[haircommander]

Does this happen when you start kubelet after crio?

we've seen this in the past when crio-status info endpoint didn't send the correct storage root, which doesn't seem to be the case. Also, your crio config looks correct (it's not changing the value of the crio config).

Just wondering, what's in /etc/containers/storage.conf ?

[nyxi]

I haven't implemented any specific startup order for kubelet / cri-o, just using the default systemd services. For cri-o that is the systemd service shipped with the package for CentOS 8 Stream.

For the node above I can see that kubelet started 10 seconds before cri-o.

My /etc/containers/storage.conf with all comments removed:

[storage]
driver = "overlay"
runroot = "/var/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
]
[storage.options.overlay]
mountopt = "nodev,metacopy=on"
[storage.options.thinpool]

I haven't made any changes to this file so it should be the default one shipped with Fedora CoreOS.

[haircommander]

what happens when you make the kubelet depend on cri-o on startup? does this error pop up?

[nyxi]

After some testing it appears that the error message in kubelet and the error events for the node go away when crio is started before kubelet.

And the vice versa is also true, the error consistently appears when crio is started after kubelet.

So my fix for now is a systemd override for crio.service containing:

[Unit]
Before=
Before=kubelet.service

[haircommander]

awesome to hear! I was worried kubelet never registers the images filesystem if it starts before cri-o. Sounds like this issue is mostly cosmetic, and there's a work around. Are we good to close this?

[nyxi]

I was worried kubelet never registers the images filesystem if it starts before cri-o

As far as I can tell this is the problem I am seeing, kubelet never registers the images filesystem if it starts before cri-o.

You probably want to fix this in the systemd service for crio that you bundle in the OS packages. Or at least document it prominently as a known issue in the installation instructions.

This caused our nodes to run out of disk space, yet Kubernetes didn't mark the nodes as having DiskPressure. I assume because kubelet incorrectly registered the images filesystem. So our clusters were trying to schedule pods on nodes that didn't have free disk to pull the images.

[haircommander]

ah I see! I would think kubelet should be able register it if it starts before cri-o, but we can work around in cri-o for now

[artheus]

[Unit]
Before=
Before=kubelet.service

@nyxi Is it really necessary to have an empty Before= statement, or is it simply a copy&paste thing?
I got quite confused by this, and I've been reading the SystemD Unit documentation, but cannot find anything about how an empty Before= would have any effect at all.

[icpenguins]

@nyxi could you elaborate on how to implement the service order fix you suggested above? I am seeing this on both the nodes that I just built with cri-o. I'm assuming you mean something like the following but would like validation if possible 👍 I am also curious to @artheus question above if the additional, blank Before= is required?

sudo vi /usr/lib/systemd/system/crio.service

# Add the following under [Unit]
Before=kubelet.service

[haircommander]

yes that should fix the problem @icpenguins

[plevart]

Encountered the same issue. As a workaround until it gets fixed in the crio package itself, I would recommend not editing the file of the crio package because updating/upgrading the package would then install .rpmnew file instead of overwriting it with new version and either way it is not what you actually want.

You can issue the following command:

systemctl edit crio

and then paste the following content into the editor and save it:

[Unit]
Before=kubelet.service

This would create new file: /etc/systemd/system/crio.service.d/override.conf with above contntent which contains just overrides to upstream crio.service so updates/upgrades of crio package are able to update that file without problems.

[haircommander]

This was (finally) fixed by #4443 upstream. the related backports, in addition to new packaged versions, will fix them downstream