Embed cri-dockerd in k3s

[ibuildthecloud]

Is your feature request related to a problem? Please describe.

Bring back --docker

Describe the solution you'd like

I'd prefer k3s to vendor in cri-dockerd similar to all go stuff and run it when --docker is requested

Describe alternatives you've considered

Additional context

Docker is the number one container tool for development and I don't see that changing anytime soon. Running k3s with --docker gives you the very nice advantage of being able to run images from docker build with no push/pull cycle.

Backporting

• Needs backporting to older releases

[brandond]

I proposed doing this for 1.24 and got voted down. cri-dockerd also turned out to have a bunch of performance issues related to pulling stats that we had to call in some favors at Mirantis to get addressed. They apparently have only one person on cri-dockerd maintenance who only checks the issues/PRs when he gets around to it: Mirantis/cri-dockerd#38 (comment)

I did get as far as actually implementing this before getting voted down though, and it worked fine: 6feaec2

[ibuildthecloud]

Is there more context as why it got voted down. Is Rancher Desktop not going to use cri-dockerd?

[brandond]

I think folks were mostly just ready to make a clean break from it at the same time as upstream did. With docker support on windows evaporating and cri-dockerd maintenance from Mirantis being so half-hearted it didn't seem worth carrying.

Rancher Desktop is using it. They are actually packaging their own docker and containerd versions, along with a fork of cri-dockerd.

[ibuildthecloud]

Basically, if Rancher Desktop feels the need to package/fork/maintain cri-dockerd that basically proves there's value in doing that work upstream because docker is the primary solution on the desktop. So can we just shift the effort of maintaining it in Rancher Desktop and do it in k3s.

[ibuildthecloud]

Who are the gods of k3s that I need to convince that this is in fact a good idea to include cri-dockerd. Now that 1.24 is the stable release this is becoming more of a problem for me.

[ekristen]

I agree this needs to come back. Docker is still the preeminent way to do container/image development not having this feature in k3s makes using k3s for development difficult and hurts its place in the ecosystem.

[phillipsj]

I believe @Oats87 is working on an issue related to cri-dockerd and CNI issues for 1.24 RKE. I would assume the issue could also exist here.

[brandond]

6feaec2 should still apply mostly cleanly, it's just a matter of PM (or the team) deciding that it's something we want to take on as a support burden.

[jandubois]

Rancher Desktop is using it. They are actually packaging their own docker and containerd versions, along with a fork of cri-dockerd.

The fork was only temporary while upstream was not applying PRs. We are bundling the latest upstream version of cri-dockerd now.

For Rancher Desktop it will not matter if k3s starts bundling cri-dockerd or not. Since we support running all older versions of k3s, we will have to continue to ship it to support e.g. 1.24.1 to 1.24.3, which don't include it.

[xiaods]

everyone like docker, but everyone care when the time switch to contained. in my side project, I use nerdctl + contained in my k3s branch. many user complain why use nerdctl. they also like docker. this is true story.
so I agree with embed cri-dockerd in k3s

[brandond]

Anyone interested in this, please 👍🏻 the issue, and try out the linked PR to see if it meets your needs.

[cwayne18]

@brandond once Derek's ADR reorg branch is merged, I think this is a perfect opportunity for an ADR to get public comment on

[brandond]

ADR has been added to the PR.

[evol262]

I proposed doing this for 1.24 and got voted down. cri-dockerd also turned out to have a bunch of performance issues related to pulling stats that we had to call in some favors at Mirantis to get addressed. They apparently have only one person on cri-dockerd maintenance who only checks the issues/PRs when he gets around to it: Mirantis/cri-dockerd#38 (comment)

I blame Github more than anything for this, since it decided not to send me notifications or emails about... anything for a while, including PRs, so the fact that there was even an active base of people using/testing it that close to 1.24 was pretty surprising. I check it much more regularly now, and I'm not the only person (in theory).

@Oats87 there was actually some work around CNI in 1.24 specifically last week. Mostly in better defaults so --network-plugin=cni isn't needed anymore (since CNI is the default in 1.24, basically), plus re-implementing SupportsSingleFileMapping for Windows after upstream k8s unceremoniously dropped it, so it works there also on 1.24.

It's not really obvious to me whether this is still a concern on k3os/RancherOS anymore, but there's at least one cri-dockerd user who's using k3s, and whatever it's doing to remap these paths into pods makes --network-plugin=no-op actually behave as expected (assuming that Flannel's setup finds the volume maps in the pods), but all of the config mangling is no fun for anyone, and with 1.24 out "for real" and picking up steam, I'm not expecting any more changes to cri-dockerd for 1.24.

The very first thing to do next (for 1.25, before, or whenever) is this, and I'd love to get more input about requirements. I can define requirements based loosely on what other CRIs do, but the input would be very welcomed.

[brandond]

better defaults so --network-plugin=cni isn't needed anymore (since CNI is the default in 1.24, basically)

At least on the version of cri-dockerd we're about to ship, not using --network-plugin=cni resulted in pods using the default docker address space, instead of the cluster pod CIDRs. When did that change?

I will note that there are a couple extra commits from me on that branch; I'm not sure if any of them would be of any use to upstream?

[evol262]

better defaults so --network-plugin=cni isn't needed anymore (since CNI is the default in 1.24, basically)

At least on the version of cri-dockerd we're about to ship, not using --network-plugin=cni resulted in pods using the default docker address space, instead of the cluster pod CIDRs. When did that change?

I will note that there are a couple extra commits from me on that branch; I'm not sure if any of them would be of any use to upstream?

A little over a week ago, with v0.2.5.

Any and all patches are welcome :) The project was in a twilight zone for a little while, since upstream extracted it and handed it over, then decided to keep it in k8s anyway, so time was spent in significantly refactoring it to make more sense as a standalone repo (restructuring the modules, not importing huge portions of k8s.io for a couple of constants which could be extracted, etc). Then github had an issue where I did not get any notifications for anything on any repo right around the time that 1.24 went live, so unless I manually refreshed the page every day, there was no way to tell what was happening.

We're on a pretty regular release cadence now, the "chicken with its head cut off" reactive development to "upstream said they were going to remove this but actually didn't, so we need a fix" or "upstream changed this, so..." is over, and most of that was around "CNI by default".

Nobody likes wrangling command line args, so adding a config file is a necessary change which makes everyone's lives easier, and Hyrum's Law makes it hard to know which flags people actually need until they're unceremoniously removed/changed and someone files an issue. I'm very much aiming for "less v0.x so the API may break" (the command-line args are more or less the API in this case) and some measure of "real" deprecation.

cgroupsv2 are an obvious, and necessary target. User namespaces are something we'll need, but the story around that with upstream moby/moby is a little muddy, and I'll probably have to do some work there. docker supports user ns remapping, but on a global level, and it requires restarting the Docker daemon, which really isn't in line with what k8s is aiming for.

On the one hand, I've spent 15 years as an open source developer, and I 💯 understand that "we need this right now" or "we need to maintain our own fork so we can backport fixes into stable branches" is a simple fact of life. On the other hand, I've always been "upstream first", and I'd love to see cri-dockerd avoid the mariadb/mysql, libreoffice/openoffice, xfree86/xorg forking debacle, even if it will never see nearly the kind of mass adoption any of those did.

[brandond]

Thanks for the commentary! I'll take a look at updating the embedded version for our next release cycle, and trying to push up anything that hasn't already been addressed. I think the major things were bumping the Kubernetes version and adding cgroupv2 support, but neither of those are huge lifts. The change to panic instead of fatal is mostly just so that we can rescue out of some (if not all) errors.

[mateuszdrab]

As we're on this topic, thought I'd sneak in a question related to supporting alternative runtimes.

I saw this article https://opni.io/v0.4/setup/gpu/#rke and it confirmed my fears that alternative runtimes are not supported with dockershim.

Do you guys know if the cri-dockerd implementation supports them?

I'm currently running k3s with --docker flag and had some issues when moving away from it so would probably need to stick with it currently.

[ShylajaDevadiga]

Validated on master branch using commit id b14cabc

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 20.04

Cluster Configuration:
One server One agent setup

To validate:
Server

sudo apt update
sudo apt install -y docker.io
curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=b14cabc107e848ea4122e6917994925b1ca2418c sh -s - server --token <TOKEN> --docker

Agent

sudo apt update
sudo apt install -y docker.io
curl -fL https://get.k3s.io | INSTALL_K3S_COMMIT=b14cabc107e848ea4122e6917994925b1ca2418c sh -s - agent --token <TOKEN> --docker --server <SERVER>:6443

Output:

ubuntu@ip-172-31-19-63:~$ kubectl get nodes
NAME              STATUS   ROLES                       AGE    VERSION
ip-172-31-19-63   Ready    control-plane,etcd,master   18m    v1.24.3+k3s-b14cabc1
ip-172-31-29-60   Ready    <none>                      3m7s   v1.24.3+k3s-b14cabc1
ubuntu@ip-172-31-19-63:~$ kubectl get pods -A 
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-b96499967-t4djb                   1/1     Running     0          25m
kube-system   helm-install-traefik-crd-2xv6k            0/1     Completed   0          25m
kube-system   helm-install-traefik-txh59                0/1     Completed   2          25m
kube-system   local-path-provisioner-7b7dc8d6f5-fskz9   1/1     Running     0          25m
kube-system   metrics-server-668d979685-srnm6           1/1     Running     0          25m
kube-system   svclb-traefik-b175a1a6-brlxk              2/2     Running     0          25m
kube-system   svclb-traefik-b175a1a6-t29d6              2/2     Running     0          10m
kube-system   traefik-7cd4fcff68-2th6c                  1/1     Running     0          25m
ubuntu@ip-172-31-19-63:~$ sudo docker images
REPOSITORY                         TAG                    IMAGE ID       CREATED         SIZE
rancher/klipper-helm               v0.7.3-build20220613   38b3b9ad736a   2 months ago    239MB
rancher/mirrored-library-traefik   2.6.2                  72463d8000a3   4 months ago    102MB
rancher/mirrored-coredns-coredns   1.9.1                  99376d8f35e0   5 months ago    49.5MB
rancher/klipper-lb                 v0.3.5                 dbd43b6716a0   6 months ago    8.09MB
rancher/local-path-provisioner     v0.0.21                fb9b574e03c3   8 months ago    35MB
rancher/mirrored-metrics-server    v0.5.2                 f73640fb5061   9 months ago    64.3MB
rancher/mirrored-pause             3.6                    6270bb605e12   11 months ago   683kB
ubuntu@ip-172-31-19-63:~$ 

Cluster Configuration:
Multi server One agent setup using etcd backend

ubuntu@ip-172-31-19-63:~$ kubectl get nodes
NAME               STATUS   ROLES                       AGE   VERSION
ip-172-31-18-87    Ready    control-plane,etcd,master   12m   v1.24.3+k3s-b14cabc1
ip-172-31-19-63    Ready    control-plane,etcd,master   14m   v1.24.3+k3s-b14cabc1
ip-172-31-21-236   Ready    control-plane,etcd,master   12m   v1.24.3+k3s-b14cabc1
ip-172-31-29-60    Ready    <none>                      12m   v1.24.3+k3s-b14cabc1
ubuntu@ip-172-31-19-63:~$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-b96499967-mgmzn                   1/1     Running     0          14m
kube-system   helm-install-traefik-crd-gwrnp            0/1     Completed   0          14m
kube-system   helm-install-traefik-n6w8n                0/1     Completed   2          14m
kube-system   local-path-provisioner-7b7dc8d6f5-trssn   1/1     Running     0          14m
kube-system   metrics-server-668d979685-6brdw           1/1     Running     0          14m
kube-system   svclb-traefik-039695c5-55wv9              2/2     Running     0          12m
kube-system   svclb-traefik-039695c5-dgvz9              2/2     Running     0          13m
kube-system   svclb-traefik-039695c5-j7hnl              2/2     Running     0          12m
kube-system   svclb-traefik-039695c5-xcfw6              2/2     Running     0          12m
kube-system   traefik-7cd4fcff68-v47rk                  1/1     Running     0          13m
ubuntu@ip-172-31-19-63:~$ sudo docker images
REPOSITORY                         TAG                    IMAGE ID       CREATED         SIZE
rancher/klipper-helm               v0.7.3-build20220613   38b3b9ad736a   2 months ago    239MB
rancher/mirrored-library-traefik   2.6.2                  72463d8000a3   4 months ago    102MB
rancher/mirrored-coredns-coredns   1.9.1                  99376d8f35e0   5 months ago    49.5MB
rancher/klipper-lb                 v0.3.5                 dbd43b6716a0   6 months ago    8.09MB
rancher/local-path-provisioner     v0.0.21                fb9b574e03c3   8 months ago    35MB
rancher/mirrored-metrics-server    v0.5.2                 f73640fb5061   9 months ago    64.3MB
rancher/mirrored-pause             3.6                    6270bb605e12   11 months ago   683kB
ubuntu@ip-172-31-19-63:~$ 

[ShylajaDevadiga]

Validated using rc v1.24.4-rc1+k3s1
Validated passing --docker flag as well as config.yaml

ubuntu@ip-172-31-19-63:~$ cat /etc/rancher/k3s/config.yaml 
token: "<TOKEN>"
docker: true
cluster-init: true

Cubuntu@ip-172-31-19-63:~$ k3s -v
k3s version v1.24.4-rc1+k3s1 (c3f830e9)
go version go1.18.1
ubuntu@ip-172-31-19-63:~$ kubectl get nodes
NAME               STATUS   ROLES                       AGE   VERSION
ip-172-31-18-87    Ready    control-plane,etcd,master   32m   v1.24.4-rc1+k3s1
ip-172-31-19-63    Ready    control-plane,etcd,master   33m   v1.24.4-rc1+k3s1
ip-172-31-21-236   Ready    control-plane,etcd,master   32m   v1.24.4-rc1+k3s1
ip-172-31-29-60    Ready    <none>                      33m   v1.24.4-rc1+k3s1
ubuntu@ip-172-31-19-63:~$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-b96499967-9cd55                   1/1     Running     0          33m
kube-system   helm-install-traefik-crd-khsmp            0/1     Completed   0          33m
kube-system   helm-install-traefik-zsnjf                0/1     Completed   2          33m
kube-system   local-path-provisioner-7b7dc8d6f5-lrtjw   1/1     Running     0          33m
kube-system   metrics-server-668d979685-8dnf5           1/1     Running     0          33m
kube-system   svclb-traefik-c12dcbe5-84cwf              2/2     Running     0          32m
kube-system   svclb-traefik-c12dcbe5-lp77k              2/2     Running     0          32m
kube-system   svclb-traefik-c12dcbe5-qdkb2              2/2     Running     0          32m
kube-system   svclb-traefik-c12dcbe5-tfpsg              2/2     Running     0          32m
kube-system   traefik-7cd4fcff68-pgnd4                  1/1     Running     0          32m
ubuntu@ip-172-31-19-63:~$ sudo docker images
REPOSITORY                         TAG                    IMAGE ID       CREATED         SIZE
rancher/klipper-helm               v0.7.3-build20220613   38b3b9ad736a   2 months ago    239MB
rancher/mirrored-library-traefik   2.6.2                  72463d8000a3   4 months ago    102MB
rancher/mirrored-coredns-coredns   1.9.1                  99376d8f35e0   5 months ago    49.5MB
rancher/klipper-lb                 v0.3.5                 dbd43b6716a0   6 months ago    8.09MB
rancher/local-path-provisioner     v0.0.21                fb9b574e03c3   8 months ago    35MB
rancher/mirrored-metrics-server    v0.5.2                 f73640fb5061   9 months ago    64.3MB
rancher/mirrored-pause             3.6                    6270bb605e12   11 months ago   683kB

[evol262]

As we're on this topic, thought I'd sneak in a question related to supporting alternative runtimes.

I saw this article https://opni.io/v0.4/setup/gpu/#rke and it confirmed my fears that alternative runtimes are not supported with dockershim.

Do you guys know if the cri-dockerd implementation supports them?

I'm currently running k3s with --docker flag and had some issues when moving away from it so would probably need to stick with it currently.

@mateuszdrab cri-dockerd is, essentially, a refactor of the dockershim driver (which the k8s team extracted and handed over). There is very recent support for alternative runtimes in docker, but it hasn't really been tested thoroughly yet. What's the use case? Can you open an issue on the cri-dockerd repo?

[mateuszdrab]

As we're on this topic, thought I'd sneak in a question related to supporting alternative runtimes.

I saw this article https://opni.io/v0.4/setup/gpu/#rke and it confirmed my fears that alternative runtimes are not supported with dockershim.

Do you guys know if the cri-dockerd implementation supports them?

I'm currently running k3s with --docker flag and had some issues when moving away from it so would probably need to stick with it currently.

@mateuszdrab cri-dockerd is, essentially, a refactor of the dockershim driver (which the k8s team extracted and handed over). There is very recent support for alternative runtimes in docker, but it hasn't really been tested thoroughly yet. What's the use case? Can you open an issue on the cri-dockerd repo?

I totally didn't realize how new support for alternative runtimes is in Docker. I could swear I tried to find a runtime flag a few weeks ago and couldn't, this explains it.

To be honest I don't really have a use case to stick with dockershim... I need to try reconfiguring my setup to run without the flag, but I tested it on one of the hosts and had issues.

In a way I don't want to run two instances of containerd, so I might need to configure K3s to use an external containerd socket.

I'll test it out more when I have some more time...