Document container runtimes available in minikube

[afbjorklund]

We should add some more detailed documentation, about the available container runtimes.

In alphabetical order:

• "containerd"

  • server: containerd + buildkitd
  • client: nerdctl (+ ctr + buildctl)

• "cri-o"

  • server: crio + podman
  • client: podman

• "docker"

  • server: dockerd
  • client: docker

We need some more information about tools and envionment, than what upstream provides:

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

Some information is available under https://minikube.sigs.k8s.io/docs/handbook/pushing/

• nerdctl env: note: nerdctl doesn't have any remote API yet, only a CLI

• podman env: https://minikube.sigs.k8s.io/docs/commands/podman-env/

• docker env: https://minikube.sigs.k8s.io/docs/commands/docker-env/

Spelling should be the same as what Kubernetes uses, when listing the runtime for a node:

minikube kubectl -- get nodes -o wide

Don't need so much information about crictl, that can be left for upstream to document.

https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/

[afbjorklund]

For containerd, see:

• Add nerdctl command for users outside kubernetes #13252

For some context, see:

• Make default container runtime dynamic #13250

[sftim]

I'm not quite clear on what's meant by the “client” part of the issue description.

Let's say I run Minikube with qemu + kvm VM driver. I don't need to know about docker, nerdctl or podman to do that.

I recommend mentioning some additional context for this issue so it's clear what kind of pages this is relevant to and under what circumstances the user's choice of client becomes relevant.

[afbjorklund]

The "client" refers to the program that runs on host, such as kubectl.

Some things can be performed by node tools like crictl, but not everything.

For instance for loading images from tarball, or building new images.

Then you have to drop down to the real container runtime, or use a wrapper.

---

Linux users can use the tools locally, but Mac and Win users are not so lucky.

What we are trying to avoid, is the scenario where the user runs two VMs.

And then have to push and pull the images between those separate machines.

Many minikube users are used to manipulating directly, using ssh and docker.

[afbjorklund]

Also, there is even a new feature where people use minikube without Kubernetes.

https://minikube.sigs.k8s.io/docs/faq/#can-i-use-minikube-as-a-docker-desktop-replacement

Previously they could use "Docker Machine", but that project has been deprecated.

It runs the VM provisioner (CR install), but does not run the bootstrapper (now: kubeadm)

---

Historic context: https://kubernetes.io/blog/2016/09/how-we-made-kubernetes-easy-to-install/

There are three stages in setting up a Kubernetes cluster, and we decided to focus on the second two (to begin with):

Provisioning : getting some machines
Bootstrapping : installing Kubernetes on them and configuring certificates
Add-ons : installing necessary cluster add-ons like DNS and monitoring services, a pod network, etc

And of course also https://kubernetes.io/blog/2016/07/minikube-easily-run-kubernetes-locally/

When the cluster no longer runs a Docker daemon, the client needs to be changed accordingly ?

[afbjorklund]

The preferred driver for minikube even runs the cluster in a container inside a virtual machine:

• Running minikube on minikube #13434

It is possible that the host no longer runs Docker Desktop, and the cluster no longer runs Docker Engine.

The subscription changes* and interface changes* are only distantly related to this, but accelerates adoption.

• https://www.docker.com/blog/updating-product-subscriptions/

• https://kubernetes.io/blog/2020/12/02/dockershim-faq/

[afbjorklund]

Let's say I run Minikube with qemu + kvm VM driver. I don't need to know about docker, nerdctl or podman to do that.

You can run minikube start, and then minikube kubectl and minikube dashboard without learning the "internals".

This was more advanced documentation, for users that want to talk directly to the container runtime (outside of the CRI)
Either because there are things missing in the specification, or because they are doing something outside of Kubernetes ?

But there should not be any need to do it, hopefully minikube image build covers that - until kubectl build is standard.
(the purpose of the "build" command is to convert a Dockerfile/Containerfile into something that can used as an image)

• https://github.com/vmware-tanzu/buildkit-cli-for-kubectl

Many things can also be accomplished with crictl only.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.