Clarity around dockershim and CRI

[dghubble]

Due to some sensationalistic messaging from upstream tweeters, there is a need to reassure end-user folks about docker.

Kubernetes uses the Container Runtime Interface (CRI) to interface with the container runtimes (dockershim+dockerd, containerd, cri-o). By default, the Kubelet provides a "dockershim" to interface with the dockerd daemon (which itself doesn't implement CRI). "Docker" consists of both the dockerd daemon and containerd (which implements most of the core bits of what folks think of as docker) which is both a subcomponent and also a CRI implementation. Fedora CoreOS and Flatcar Linux provide dockerd (and therefore containerd as well).

In v1.20, Kubelet will emit a warning log about the dockershim being removed in the future (possibly v1.22). This is intended as a message to distro authors, not end-users. Kubernetes will continue to run container images, you can continue to build/develop container images with docker, write Dockerfiles, etc. End-users can stop reading/worrying now. The upstream Kubernetes motive is that it is simpler to just call containerd directly, rather than dockershim+dockerd (which then uses containerd).

Typhoon (as of v1.20.4) uses the Kubelet default dockershim wrapping the host's dockerd, will continue to in v1.20, and will internally switch to calling containerd when it is safe to do so. However, the CRI interface (alpha2) is still evolving toward v1 (kubernetes/kubernetes#96325 (comment)) and OSes will need to ship new containerd releases for that event. Unlike Kubernetes bundling "dockershim" to handle compat with many DockerCE versions, Kubernetes CRI must be kept compatible with the host containerd CRI version, so it is advantageous to wait until these spec versions are sorted out.

• Kubernetes CRI version stabalizes some
• Host OSes provide newer containerd (or cri-o)
  • Flatcar Linux
  • Fedora CoreOS
• Kubernetes updates min CRI version conservatively (host containerd must stay ahead)

Related:

• Flatcar Linux
• Fedora CoreOS

[justaugustus]

ref an inbound FAQ Kubernetes is planning to publish: kubernetes/website#25360

[dghubble]

Thanks Stephen

[justaugustus]

Anytime, bud! 💕

[justaugustus]

Also, sent a note to the community to chat about how we can improve comms in future: https://groups.google.com/g/kubernetes-dev/c/B0mPMpDLVk4

[travier]

I think you meant this link: coreos/fedora-coreos-tracker#767 for the Fedora CoreOS tracking issue.

[anthr76]

Considering Fedora CoreOS now has rpm-ostree layering support for modules. Is there any interest to ship typhoon with cri-o for Fedora CoreOS? a 1.22 module exists https://src.fedoraproject.org/modules/cri-o/tree/1.22 and seems to be being maintained from this point forward.

[dghubble]

Flatcar Linux will move to containerd in the next release.

[log1cb0mb]

Flatcar Linux will move to containerd in the next release.

@dghubble Not yet for Fedora CoreOS?
I would like to mention that I am already using cri-o with manual cri-o package installation and things seem to work. Or I am not yet completely sure if I have tested everything.
May be I should move to containerd instead, I am unable to decide. And I suppose if I have to think about deciding between then I shouldnt even bother :/

Edit: Allright, this coreos/fedora-coreos-tracker#767 (comment) kind of answers my question but specifically for cri-o

[dghubble]

Yeah, there is still a large gap between getting cri-o to work manually and it being ready for production use coreos/fedora-coreos-tracker#767 (comment) I'm not happy with its state on FCOS so we wait.

[dghubble]

Hey folks. Typhoon for Fedora CoreOS will soon move from docker-shim to containerd (similar to Typhoon for Flatcar Linux). I know this will disappoint some cri-o engineers, so I wanted to provide a bit of detail.

Background

Kubernetes uses the Container Runtime Interface (CRI) to interact with container runtimes (docker-shim, containerd, cri-o). The CRI originated out of early efforts to decouple Kubernetes from docker to allow using CoreOS rkt (defunct) and other runtimes, around the time containers were standardized into OCI.

The pluggable abstraction was a success. Kubernetes decoupled itself from Docker, while docker-shim allowed docker (which was shipped by most OSes and fairly stable) to be used. The focus is now on Kubernetes concepts, rather than specific underlying container runtimes as much.

Typhoon has used docker-shim reliably for a number of years, with Fedora CoreOS and Flatcar Linux keeping docker updated. Its been unremarkable, which is good. However, since Kubernetes v1.20 announced docker-shim's eventual removal, I've been evaluating the right choice for a CRI compatible container runtime.

Kubernetes v1.24

Kubernetes v1.24 will remove support for docker-shim from the Kubelet to shrink the Kubelet. Its now time to shift the underlying container runtime and work out any troubles.

Fedora CoreOS

Fedora CoreOS (FCOS) is a container optimized distro. Today FCOS ships podman and docker (and consequently containerd). Last year I raised an issue to discuss how cri-o or containerd could be "officially" supported on Fedora CoreOS. But a decision now needs to be made.

cri-o

Red Hat emphasizes cri-o as their favored container runtime. I know Red Hat engineers have put some great efforts into cri-o and my intent was for FCOS to use cri-o and Flatcar to use containerd. After all, Fedora should be perfectly positioned to make Red Hat's own cri-o software a first class citizen on Fedora CoreOS.

However, the decision is about the holisitic experience and about what I trust to run my own clusters and what's best for the distro.

Concerns

• Pragmatics: FCOS 35.x cannot currently manually install cri-o v1.23 because of Fedora 35 not shipping Go v1.17. If had adopted cri-o months ago, we'd be in a real pickle. Fedora modules may eventually improve this and unblock the iteration velocity.
• Package Philosophy: cri-o uses a strict versioning scheme (e.g. cri-o v1.23.0 supports Kubernetes v1.23.0). Typhoon ships Kubernetes within days of release. Waiting on upstream packaging breaks this release cadence
• Butane Config: No great way to layer in packages "for real". Can rpm-ostree ex module install in a script unit, but that still has the feel of breaking the warranty (i.e. how officially will you live be supported).
• Docs feel incomplete
• OpenShift gets cri-o because its bundled into RHEL CoreOS. See Clarity around dockershim and CRI #899 (comment) It curls down and runs a script (doesn't checksum either)

Thematically, I'm sad to see cri-o still mired in packaging. But even if we ignore RPM/modularity anything, grit our teeth, and bundle cri-o into init container images to drop on hosts (CNI plugin style) or write fetch scripts, the rigid compatibility would still leave me delaying Kubernetes releases, waiting on https://github.com/cri-o/cri-o/releases.

containerd

Fedora CoreOS implicitly already ships containerd by shipping docker. Just enable containerd.service and write the /etc/containerd/config.toml file and it works. It worked over version skews as well.

Concerns

• Unknowns: Fedora CoreOS may later try to remove containerd as well as docker
• Compatibility: No real guarantees that Fedora CoreOS's containerd will stay new enough. However, Kubernetes is likely to move slowly with regard to min version incements and CoreOS will keep bumping containerd (at least for CVEs)

Mitigating Factors

• containerd has real world utility and is independent of docker. Red Hat could achieve its goals by just removing docker and leaving containerd
• When docker is removed from FCOS, if containerd is included I'd assume there will be a migration path that forces the existence of a containerd module and a Butane config that could restore it

Future

On Fedora CoreOS, containerd works. It works now, without an install or fuss. It has decent docs. And it passes Kubernetes conformance. At this point I believe the best thing for the distro is to adopt containerd and ready ourselves for Kubernetes v1.24.

Flatcar Linux

Flatcar Linux officially supports both docker and containerd in order to make the switch. Its has some nice docs on the switch. containerd is the clear choice and became the container runtime on Flatcar Linux in Typhoon v1.23.1.

Notes

• Using containerd on Fedora CoreOS its not a forward promise. If conditions change and cri-o improves, Typhoon for Fedora CoreOS could switch to it. Users should continue to treat the underlying container runtime as an internal implementation detail
• Those who do hacky things like mounting docker.sock inside Pods will of course be impacted by docker being disabled (and I'd predict, eventually removed by OS vendors). I'm not aiming to address that.

[haircommander]

Thanks for providing clarity in your position and clearly stating the reasoning. While I'm disappointed about the decision as it stands, I hope to one day have the CRI-O packaging in a position that would cause you to reconsider.

But even if we ignore RPM/modularity anything, grit our teeth, and bundle cri-o into init container images to drop on hosts (CNI plugin style) or write fetch scripts, the rigid compatibility would still leave me delaying Kubernetes releases, waiting on https://github.com/cri-o/cri-o/releases.

Obviously there is a lot of work to do before this, but I would consider pushing the CRI-O community to release in a more timely manner--possibly even defining a SLA so we're guaranteed to release within e.g. 3 days of a kubernetes release.

[LorbusChris]

* OpenShift itself doesn't install a `cri-o` package. It [curls](https://github.com/cri-o/cri-o/blob/main/scripts/node_e2e_installer#L18) down and runs a script (doesn't checksum either)

Small correction: This isn't the case. OpenShift (OCP) ships cri-o as part of RHEL CoreOS (and the same is currently the case for OKD, where FCOS is rebased to an OKD-specific rpm-ostree that includes cri-o):
https://github.com/openshift/os/blob/ebae3c36c63bb1180eacb60b48910bc7daa823d9/manifest.yaml#L213
I think the linked script is only used for cri-o testing.

[dghubble]

@LorbusChris thanks for the correction on how RHEL CoreOS / OKD include cri-o.