containerd config is overwritten on minikube restart

[lianmakesthings]

ctlptl is possibly doing something unsupported to configure the containerd config, which minikube overrides on restart.

Reported on Slack: https://kubernetes.slack.com/archives/CESBL84MV/p1642175814004400

I'm having an issue with minikube wiping out the config patch that ctlptl applies to connect to the registry after stopping and starting the cluster. Here is what my /etc/containerd/config.toml looks like after restarting minikube:

    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.diff-service]

[TimothyLoyer]

I found this by cat'ing the containerd config after repeated problems from linux users of our local dev environment. It will be working and then we'll start getting ImagePullBackoff errors from Kubernetes. Usually this occurs after a reboot or cluster stop and start, but I've had it happen seemingly out of the blue as well.

Using this command to cat minikube ssh cat /etc/containerd/config.toml | grep 'plugins.cri.registry' -A 1 I can confirm that the containerd config looks correct after a fresh install:

    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."localhost:5002"]
          endpoint = ["http://ctlptl-registry:5000"]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]

After minikube stop; minikube start I see this:

    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]

Here are my environment details:
Tilt 0.23.5
ctlptl 0.7.0
Minikube v1.24.0, VM Driver: docker, Runtime: containerd
K8s client: v1.22.1
K8s server: v1.22.3

[nicks]

to be clear - users are running minikube stop and minikube start manually?

This might not be something we can possibly fix. Once the cluster is set up, no ctlptl code is running, and if you run a command that borks your cluster, there's not much we can do about that.... We might have to file a bug upstream with minikube? i don't what sorts of guarantees minikube stop / minikube start makes around preserving the existing VM/container runtime...

[TimothyLoyer]

to be clear - users are running minikube stop and minikube start manually?

Good point, it may be worth switching them to using minikube pause/unpause. I'll test it out, unsure whether that solves the reboot issue though.

For now I have them using part of the old Tilt minikube-local script to re-patch the config:

#!/bin/bash
​
reg_host="ctlptl-registry"
reg_port="5002"
​
# Below code is lifted from https://github.com/tilt-dev/minikube-local
​
# patch the container runtime
# this is the most annoying sed expression i've ever had to write
minikube ssh sudo sed "\-i" "s,\\\[plugins.cri.registry.mirrors\\\],[plugins.cri.registry.mirrors]\\\n\ \ \ \ \ \ \ \ [plugins.cri.registry.mirrors.\\\"localhost:${reg_port}\\\"]\\\n\ \ \ \ \ \ \ \ \ \ endpoint\ =\ [\\\"http://${reg_host}:5000\\\"]," /etc/containerd/config.toml
​
# restart the container runtime
minikube ssh sudo systemctl restart containerd
​```

[nicks]

Was thinking about it over the weekend, and one option is simply to stop using registry mirroring.

Mirroring makes it so that the image has the same URL both inside and outside the cluster. (e.g., localhost:5000/my-image)

But you don't need it. Without mirroring, you push your image to localhost:5000/my-image. But if you want to use it in the cluster, you use the name 'ctlptl-registry:5000/my-image` (or some DNS name for the registry from inside the cluster).

Tilt supports this mode and will do the URL correction automatically. Most clusters support mirroring because this can look very alarming/confusing to users. But it will work OK.

[TimothyLoyer]

But you don't need it. Without mirroring, you push your image to localhost:5000/my-image. But if you want to use it in the cluster, you use the name 'ctlptl-registry:5000/my-image` (or some DNS name for the registry from inside the cluster).

Tilt supports this mode and will do the URL correction automatically. Most clusters support mirroring because this can look very alarming/confusing to users. But it will work OK.

I'm not sure if this is what you were suggesting, but I've tried adding the following and it seems to have no effect in correcting the ability of the minikube cluster to reach the ctlptl-registry:

default_registry(
    "localhost:5002",
    host_from_cluster="ctlptl-registry:5000",
)

I've testing localhost:5002 as working locally, and ctlptl-registry:5000 as working via minikube ssh curl, but I still get image pull backoffs.

[nicks]

it's hard to say from here why you couldn't get it to work, but i would expect tilt to ignore default_registry if there was a registry configmap in the cluster (i.e., kubectl get configmap -n kube-public local-registry-hosting -o yaml)

[TimothyLoyer]

I found this SO which may be relevant to this issue- setting the mirror on minikube start with --registry-mirror kubernetes/minikube#6848

[nicks]

nah, minikube's start --registry-mirror refers to this: https://docs.docker.com/registry/recipes/mirror/, which is really more like a proxy

CRI registry mirrors are very different, they're really more like "alternate hosts"