Using private docker registry as a pull-through cache with k3s & containerd

[onedr0p]

Hi, I was wondering it anyone can help me try to get this to work. I cannot get containerd to use my docker registry as a pull thru cache. Here is more information:

❯ k get nodes
NAME           STATUS   ROLES    AGE   VERSION
k3s-worker-d   Ready    worker   95d   v1.18.4+k3s1
k3s-master     Ready    master   95d   v1.18.4+k3s1
k3s-worker-c   Ready    worker   95d   v1.18.4+k3s1
k3s-worker-b   Ready    worker   95d   v1.18.4+k3s1
k3s-worker-a   Ready    worker   95d   v1.18.4+k3s1
k3s-worker-e   Ready    worker   95d   v1.18.4+k3s1

I ran the following on k3s-worker-e:

$ sudo cat /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "http://192.168.42.120:5000"
  192.168.42.120:5000:
    endpoint:
      - "http://192.168.42.120:5000"

$ sudo systemctl restart k3s-agent

$ sudo crictl info
...
    "registry": {
      "mirrors": {
        "192.168.42.120:5000": {
          "endpoint": [
            "http://192.168.42.120:5000"
          ]
        },
        "docker.io": {
          "endpoint": [
            "http://192.168.42.120:5000"
          ]
        }
      },
      "configs": null,
      "auths": null
...

$ sudo ctr image pull docker.io/library/traefik:latest
docker.io/library/traefik:latest:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0991b33c566f8d05bffb182dcd664b9ea99e2d401ec2aa6d4cf2f08b5ec00388:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:cc787df94ad97b555b43614b289cbd1d27b6941ae8ab7b491ea865d9108f5ad6: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ff21389d6821d06002591248b53be0d03e47ccd1d426ff04d032bd8972f80995:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08:    exists         |++++++++++++++++++++++++++++++++++++++|
config-sha256:a9dcd182440c7b69870034f0a8083c24f2d2ffa6d7919bdbff84cbe03a5d85d1:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f16506d32a25436b458a7f443ec1ca0d096d99d3479523c92c247856fc6148c0:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:dece8d655ab616ffab2ae66dda60b30827793930bb30e38fe04fd4bce39e965f:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.4 s                                                                    total:  21.3 M (15.2 MiB/s)
unpacking linux/amd64 sha256:0991b33c566f8d05bffb182dcd664b9ea99e2d401ec2aa6d4cf2f08b5ec00388...
done

Now I checked to see if the traefik image was cached in my registry and it was not found.

$ k exec -it pod/registry-docker-registry-765f685446-kjtq2 -- sh
/ $ cd /var/lib/registry/
/var/lib/registry $ ls
lost+found

After that, I tried on my local macbook with the following Docker config and the images do get cached on my registry:

{
  "experimental": true,
  "debug": true,
  "insecure-registries": [
    "192.168.42.120:5000"
  ],
  "registry-mirrors": [
    "http://192.168.42.120:5000"
  ]
}

❯ docker pull traefik:latest
latest: Pulling from library/traefik
cbdbe7a5bc2a: Pull complete
f16506d32a25: Pull complete
dece8d655ab6: Pull complete
ff21389d6821: Pull complete
Digest: sha256:0991b33c566f8d05bffb182dcd664b9ea99e2d401ec2aa6d4cf2f08b5ec00388
Status: Downloaded newer image for traefik:latest
docker.io/library/traefik:latest

$ k exec -it pod/registry-docker-registry-765f685446-kjtq2 -- sh
/var/lib/registry/docker/registry/v2/repositories/library $ ls
traefik

Related issues: #1802 and #1210 (comment)

Is there some flaw in my testing? Should I not be using sudo ctr image pull ... as a test?

[onedr0p]

This appears to be working now, I guess the only way to test is to restart all agents and server and do not use sudo ctr image pull....

Deploy changes to your manifests and it should start working.

[brandond]

Yeah, running through the process of copying out the file to all your nodes and then restarting k3s on all of them is definitely necessary to get it to pick up the change.

I'm a little surprised that ctr wasn't honoring the mirror settings though. That could be a separate issue if confirmed. What do you get from sudo ctr --version?

[onedr0p]

$ sudo ctr --version
ctr github.com/rancher/containerd v1.3.3-k3s2

[brandond]

Hmm that seems like it might be a bug then. I am not super familiar with the internals of containerd but I would expect ctr to respect the containerd configuration.

[caroline-suse-rancher]

Closing due to age, and we seem to have a fix. Reopen if needed