Provide hooks to configure cri-dockerd via a configuration file

[evol262]

Similar to other CRI and CNI providers, the amount of possible churn in cri-dockerd in keeping up with upstream k8s may become a lot, and shifting around defaults in command-line args is not a great experience for consumers.

viper is already part of the deptree (though we'll pull a new version), and we can relatively easily expose NetworkPluginSettings, ContainerRuntimeOptions, and the small amount of daemon-specific options while keeping reasonable defaults.

Additionally, this would allow for all of these to easily be set/tweaked via environment variables if desired.

TOML is not the greatest format in the world, but it's reasonably well supported by burntsushi/toml into viper, with headings as above (daemon-specific options as top-level/without subkey).

@afbjorklund input?

[afbjorklund]

That sounds good, and similar to what the other container runtimes (containerd, cri-o) are already doing...

So it should be familiar to the kubernetes configuration documentation, for updating e.g. the pause image ?

Currenty cri-dockerd is a special case, so it even has a special section in the kubeadm documentation:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#override-pause-image-cri-dockerd

---

Migrating to the systemd cgroups driver in done in Docker, and not in the CRI layer like with the others. (?)

So I think that it doesn't need tweaking in cri-dockerd, and can refer to the Docker Engine documentation ?

e.g. https://docs.docker.com/config/containers/runmetrics/ (or wherever it is documented, searched for "v2")

Recent versions of Kubernetes have now changed from "cgroupfs" to "systemd" driver, by default.

[afbjorklund]

Typically those two were the only values tweaked, maybe with the addition of hairpin mode for IP selection.

-ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=
+ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --pod-infra-container-image={{.PauseImage}} --network-plugin={{.NetworkPlugin}}{{.ExtraArguments}}

Where the "extra arguments" could be something more special like: --hairpin-mode=promiscuous-bridge

---

^ above showing the Go template used in the code*, not the actual resulting systemd override file...

/etc/systemd/system/cri-docker.service.d/10-cni.conf

* https://github.com/kubernetes/minikube/blob/v1.26.1/pkg/minikube/cruntime/docker.go#L705