Enabled "--embedded-registry" on v1.28.7+k3s1

[reefland]

Bare metal Ubuntu 22.04.x based x86 systems.

$ k get nodes
NAME    STATUS   ROLES                       AGE    VERSION
k3s01   Ready    control-plane,etcd,master   717d   v1.28.7+k3s1
k3s02   Ready    control-plane,etcd,master   690d   v1.28.7+k3s1
k3s03   Ready    control-plane,etcd,master   697d   v1.28.7+k3s1
k3s04   Ready    worker                      463d   v1.28.7+k3s1
k3s05   Ready    worker                      463d   v1.28.7+k3s1
k3s06   Ready    worker                      391d   v1.28.7+k3s1

Trying out the new Spegel registry mirror. Set --embedded-registry flag only on the 3 control-plane nodes so far.

Created this:

cat /etc/rancher/k3s/registries.yaml

mirrors:
  docker.io:
  registry.k8s.io:
  ghcr.io:
  quay.io:
  mcr.microsoft.com:
  public.ecr.aws:
  gcr.io:
  k8s.gcr.io:
  lscr.io:  

Then:

sudo systemctl daemon-reload
sudo systemctl restart k3s

Trying to figure out if this is working correctly, not sure what to look for, so far each of the 3 nodes only show this:

$ journalctl -b -u k3s | grep -i 'p2p\|spegel'

Mar 17 18:51:03 k3s01 k3s[1736992]: time="2024-03-17T18:51:03-04:00" level=info msg="Starting distributed registry P2P node at 192.168.10.215:5001"
Mar 17 18:51:08 k3s01 k3s[1736992]: time="2024-03-17T18:51:08-04:00" level=info msg="Node P2P address annotations and labels added: /ip4/192.168.10.215/tcp/5001/p2p/QmUSRvE9ckBaWwWSLyx3o99h7Zx3sBkMp8CTNiH8LdcG14"
Mar 17 18:52:12 k3s01 k3s[1736992]: 2024-03-17T18:52:12.786-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:52:12 k3s01 k3s[1736992]: 2024-03-17T18:52:12.787-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:52:12 k3s01 k3s[1736992]: 2024-03-17T18:52:12.788-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:53:03 k3s01 k3s[1736992]: 2024-03-17T18:53:03.162-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:55:03 k3s01 k3s[1736992]: 2024-03-17T18:55:03.163-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:55:36 k3s01 k3s[1736992]: 2024-03-17T18:55:36.751-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:55:36 k3s01 k3s[1736992]: 2024-03-17T18:55:36.752-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:55:36 k3s01 k3s[1736992]: 2024-03-17T18:55:36.752-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers
Mar 17 18:57:03 k3s01 k3s[1736992]: 2024-03-17T18:57:03.162-0400        WARN        dht        go-libp2p-kad-dht@v0.25.2/lookup.go:43        network size estimator track peers: expected bucket size number of peers

Are the WARN messages anything to be worried about?

As to rolling this out on k3s-agents, they don't have a directory /etc/rancher/k3s, I assume I would need to create this and place the registries.yaml within it... but want to make sure the directory name is not k3s-agent vs k3s, etc.

Within Prometheus, I can see metrics:

[brandond]

I would start by reading https://docs.k3s.io/installation/registry-mirror and https://docs.k3s.io/installation/private-registry. The second page should answer your question about where the file goes on nodes.

All the logs and screenshots appear normal for a node running spegel.

[reefland]

Thank you for the prompt response. I obviously already read these pages in order to get it working on the control-plane. I can't get it working on the agents.

I create the directory and added the same registry list I use on the control-plane:

sudo mkdir /etc/rancher/k3s

sudo vi /etc/rancher/k3s/registries.yaml

mirrors:
  docker.io:
  registry.k8s.io:
  ghcr.io:
  quay.io:
  mcr.microsoft.com:
  public.ecr.aws:
  gcr.io:
  k8s.gcr.io:
  lscr.io:  

Set the flag on the k3s-agent such as:

ExecStart=/usr/local/bin/k3s \
    agent \
        '--embedded-registry' \
        '--kubelet-arg=image-gc-low-threshold=50' \
        '--kubelet-arg=image-gc-high-threshold=65'

$ sudo systemctl daemon-reload
$ sudo systemctl restart k3s-agent

Results in msg="flag provided but not defined: -embedded-registry":

Job for k3s-agent.service failed because the control process exited with error code.
See "systemctl status k3s-agent.service" and "journalctl -xeu k3s-agent.service" for details.

Mar 17 23:47:31 k3s04 k3s[157869]: time="2024-03-17T23:47:31-04:00" level=fatal msg="flag provided but not defined: -embedded-registry"
Mar 17 23:51:10 k3s04 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE

• I triple checked the flag has two dashes
• I tried with and without being wrapped in single-quotes, same error message

I had to remove the flag to start the agent.

[brandond]

I obviously already read these pages in order to get it working on the control-plane. I can't get it working on the agents.

https://docs.k3s.io/installation/registry-mirror#enabling-the-distributed-oci-registry-mirror

In order to enable the embedded registry mirror, server nodes must be started with the --embedded-registry flag, or with embedded-registry: true in the configuration file. This option enables the embedded mirror for use on all nodes in the cluster.

That is a server flag, and cannot be used on agents. Setting it on the server nodes enables it cluster-wide.

[reefland]

Thank you. Wasn't clear from reading the docs but makes sense.