cri-dockerd system unit override assumes /usr/bin/cri-dockerd but install instructions from mirantis point to /usr/local/bin/cri-dockerd

[lucastheisen]

What Happened?

The template for cri-dockerd systemd override uses a hard coded path:

	var CRIDockerServiceConfTemplate = template.Must(template.New("criDockerServiceConfTemplate").Parse(`[Service]
ExecStart=
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin={{.NetworkPlugin}}{{.ExtraArguments}}`))

Which differs from the install location suggested by the cri-dockerd installation instructions, specifically, this part:

install -o root -g root -m 0755 bin/cri-dockerd /usr/local/bin/cri-dockerd
cp -a packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

Attach the log file

Nov 01 16:51:50 MM269697-PC systemd[4939]: cri-docker.service: Failed at step EXEC spawning /usr/bin/cri-dockerd: No such file or directory

Operating System

Redhat/Fedora

Driver

None (Baremetal)

[lucastheisen]

You may be able to work around this by seeing what the 10-cni.service that gets generated looks like then copying it to 11-cni.service and changing the /usr/bin/cri-dockerd to /usr/local/bin/cri-dockerd. This will work as long as the other templated values dont change...

[afbjorklund]

Normally you would want to use the packages (like rpm), but as long as cri-dockerd requires horrible workarounds like hacking the ExecStart it should be able to cope with both...

It needs to read the value from any existing systemd unit, ideally it should not overwrite but replace as little as possible

[afbjorklund]

This will improve when there is a config file:

• Provide hooks to configure cri-dockerd via a configuration file Mirantis/cri-dockerd#102

[shu-mutou]

I don't really understand why the installation guide in cri-dockerd repository bothers to rewrite the installation destination, so I can't judge which is better.
So I took workaround put cri-dockerd into both of /usr/bin/ and /usr/local/bin/.

[afbjorklund]

@shu-mutou : since they changed the installation destination, they need to also change the configuration (sed).

Even the cp -a fails (with SELinux), so there are lot of issues with that "drive- by" documentation in the README.md

[afbjorklund]

Probably it should look more like https://github.com/containerd/containerd/blob/main/docs/getting-started.md

But for most people it would be better to use packages (deb/rpm), and containerd.io lets Docker handles all that...

[shu-mutou]

@afbjorklund Thank you very much for your explanations.
Neither cri-dockerd nor minikube will be used in a production environment, so I think it will only be used by developers. So it doesn't have to be a package, IMHO....

[afbjorklund]

We copy the binary and the systemd files, but it is more work (for less features) than the packages

[shu-mutou]

Finally, not like the installation guide, I putted systemd files without modification and putted cri-dockerd into /usr/bin/.

[shu-mutou]

I realized that the correct way to install cri-dockerd is not officially documented.

No matter where cri-dockerd is installed, the /etc/systemd/system/cri-docker.service.d/10-cni.conf created by minikube should use the same path as the ExecStart described in the /etc/systemd/system/cri-docker.service, I think.

[shu-mutou]

@afbjorklund
What do you think about changing as following:

1. Get path of cri-dockerd from ExecStart in /etc/systemd/system/cri-docker.service
2. Use the above in ExecStart in /etc/systemd/system/cri-docker.d/10-cni.conf

If you agree with above, could you assign me this?

[neersighted]

There is not really a way to append like you suggest (systemd/systemd#21461), unless you mean parsing the systemd unit to try to guess where cri-dockerd is -- that's kind of gross, and searching the $PATH seems cleaner.

That being said, I still want to understand where all this confusion/where all the end-users for cri-dockerd come from. It's a tool/component aimed at advanced users (e.g. those assembling their own Kubernetes) in my mind; clearly that is not reality (see Mirantis/cri-dockerd#154 (comment)), so understanding the where/why of end-users would be advantageous.

Maybe cri-dockerd just needs a ## minikube section if users of minikube are installing it all of a sudden (and are not equipped to determine the best install method themselves)?

In any case, the reason for the differing paths is simple:

• Packages are 'system managed' and should be installed to /usr/bin like anything else
• A manually copied binary is 'system administrator managed' and should be in /usr/local/bin by convention, to avoid conflicting with the package manager

This is not a hard and fast rule, but is a decades-old convention that is generally well understood. Certainly the most pragmatic thing to do in minikube seems to be searching $PATH before writing out the override unit. Alternatively, it could make a lot of sense to just table this in favor of a config file for cri-dockerd so it is not necessary to override the invocation of the daemon.

[afbjorklund]

I'll just do a similar hack, similar to the weird sudo settings of CentOS 7, and move on.

I don't really want to be parsing systemd stuff, and hope we soon get a real config file

[afbjorklund]

For my testing, I did this:

sudo apt update
sudo apt install -y conntrack socat

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

curl -LO https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.0/cri-dockerd_0.3.0.3-0.ubuntu-focal_amd64.deb
sudo dpkg -i cri-dockerd_0.3.0.3-0.ubuntu-focal_amd64.deb

I don't really like hard-coding versions, since they get outdated, but without support for apt it gets messy.

It would have been easier to install it with Docker Engine.

[afbjorklund]

@neersighted the users also need to install cri-tools and cni-plugins, which also fails in similar ways

• Install cri-tools/cni-plugins with bootstrapper, and not with provisioner #15460

[neersighted]

In all honestly, I wonder if the none driver is still useful? If the user has to fetch all the batteries that minikube doesn't concern itself with (and this is supposed to be a beginner-friendly tool for setting up a local K8s), maybe the driver has outlived its usefulness/kubeadm can substitute for this use case?

I don't use minikube much outside of kind/VM scenarios, so I guess I don't truly understand the drive for the none driver. That being said, it seems like maybe it represents a footgun these days due to the complexity of modern Kubernetes + the temptation to unsandbox Kubernetes for unsophisticated users...

[afbjorklund]

Yes, it is something being discussed. If minikube should be thrown out of the getting started tutorials...

But exposing the users to raw kubeadm is mean, and the get.k8s.io script only works with GCE nodes

[neersighted]

I don't know what the right path is here; I'm hardly objective as I've been dealing with this tooling for years, and with software engineering for even longer. That being said, I really think that none should either be useful and come with all the batteries needed (e.g. like the old hyperkube) or go away, as it's current half-useful/half-a-working-cluster state seems counter to the goal of targeting novice users.

[logopk]

@afbjorklund

For my testing, I did this:

sudo apt update
sudo apt install -y conntrack socat

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.deb
sudo dpkg -i minikube_latest_amd64.deb

curl -LO https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.0/cri-dockerd_0.3.0.3-0.ubuntu-focal_amd64.deb
sudo dpkg -i cri-dockerd_0.3.0.3-0.ubuntu-focal_amd64.deb

I don't really like hard-coding versions, since they get outdated, but without support for apt it gets messy.

It would have been easier to install it with Docker Engine.

Did that ever work due to #15413

[afbjorklund]

It works locally ("none" driver), haven't tested the "ssh" driver - but that is already broken in many other ways

kubernetes/website#33936 (comment)

[logopk]

@afbjorklund

It works locally ("none" driver), haven't tested the "ssh" driver - but that is already broken in many other ways

So you run minikube directly on the VM/Docker/Kubernetes host?

[afbjorklund]

@neersighted :

1. The Kubernetes project is currently using version 1.20.3 for their tutorials, which are "fine" up until 1.24.0 happened

2. The "none" driver is not targetting novice users, but specific use cases like online tutorials or CI pipeline tests

[logopk]

1. The Kubernetes project is currently using version 1.20.3 for their tutorials, which are "fine" up until 1.24.0 happened

Booh!

[neersighted]

I see... I'm not that qualified to comment on the maintenance situation of minikube, so I think I'll stop speculating there. Nonetheless, I think we need to figure out a better solution for everyone to the cri-dockerd situation, which might help with figuring out some of the other sharp edges.

I'm inclined to say that that is mostly docs + automation work in minikube, along with a side of "This is upstream/aimed at developers & integrators, not end users, here are some best-effort binaries" in cri-dockerd. I'd like to do more/get better than that, but summarizing the current state in a way that is explicit instead of implicit is a good first step.

[afbjorklund]

Here was the original ticket about improving libmachine API:

• Make the provisioner interface support more container runtimes #10883

But at this point, it is more about making it look good when dead:

• https://en.wikipedia.org/wiki/Lying_in_state

[neersighted]

That helps, I think I understand the lay of the land much better. I think I'll try and spend some time on this at some point. No promises as to timeline; there are a lot of plates spinning and I increasingly have less time for technical work (especially a project I have not contributed to before), but I think this is important and I'd love to provide some more hands/eyes if possible.

This is all still quite hairy, but I do have the slight advantage of having worked on two other tools that do similar things to minikube (one is the predecessor to docker-machine, still used internally at Docker and Mirantis, and the other is basically 'minikube for Mirantis Kubernetes Engine').

[afbjorklund]

The kubernetes tutorials are handled by SIG Docs, and their discussion is here: kubernetes/website#38878

https://www.oreilly.com/online-learning/leveraging-katacoda-technology.html (katacoda.com was shut down)

* Starting control plane node minikube in cluster minikube
* Running on localhost (CPUs=2, Memory=2460MB, Disk=194868MB) ...
* OS release is Ubuntu 18.04.5 LTS
* Preparing Kubernetes v1.20.2 on Docker 19.03.13 ...

[shu-mutou]

I'll let you know my use case for reference.

I'm one of the maintainers of Kubernetes Dashboard, and since 2019 I've been using a k8s cluster created on a VM using minikube and the none driver for Dashboard development.

The reasons are as follows.

• I don't like virtualbox
• At the time, minikube was said to be the default development environment for k8s.
• In order for Dashboard to support new releases of k8s, we need to prepare a new version of the k8s cluster. Minikube supports new versions of the k8s faster than kind. So I'm very grateful to use it.
• I haven't had any problems with this usage so far, and I didn't want to affect the host environment, so I haven't changed from cluster creation using the none driver on VMs.

Thanks!!

[afbjorklund]

@shu-mutou it is a valid use case, and something that should be explicitly supported by minikube

Ironically, new releases are only tested with containerd and kind - and not with docker and minikube.

---

If minikube is "replaced" in the tutorials, it will only be with a similar cloud setup calling kubeadm directly.

So you can continue to run minikube start to accomplish the same thing, hopefully with minimum effort.