Create a new single node cluster with embedded registry mirror does not work

[frederikhs]

I have followed https://docs.k3s.io/installation/registry-mirror

Environmental Info:
K3s Version:
v1.29.1+k3s1

Node(s) CPU architecture, OS, and Version:

Linux laptop-1 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

1 server

Describe the bug:

Steps To Reproduce:

• curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.29.1+k3s1" sh -s - --disable traefik --embedded-registry
• Wait until cluster is ready
• sudo k3s kubectl get daemonset -A

Expected behavior:

No spegel daemonset is created. I expected a daemonset to be created when adding --embedded-registry

Actual behavior:

No daemonset is created. Running sudo k3s kubectl get daemonset -A results in No resources found

Additional context / logs:

$ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="v1.29.1+k3s1" sh -s - --disable traefik --embedded-registry
[INFO]  Using v1.29.1+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.1+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.29.1+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

$ sudo k3s kubectl get node
NAME       STATUS   ROLES                  AGE   VERSION
laptop-1   Ready    control-plane,master   9s    v1.29.1+k3s1

$ sudo k3s kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   local-path-provisioner-84db5d44d9-z92q2   1/1     Running   0          74s
kube-system   coredns-6799fbcd5-m2dln                   1/1     Running   0          74s
kube-system   metrics-server-67c658944b-6s4t4           1/1     Running   0          74s

$ sudo k3s kubectl get daemonset -A
No resources found

[vitobotta]

I can't get the embedded registry to work either. I am using the flag for the servers, have opened port 5001 and created the registries.yaml so that all registries are mirrored, but nothing happens. There is no sign of Spegel anywhere in the cluster and the images are not being distributed between nodes. I know that for sure because the reason why I want to use it is as a workaround for a know issue with some Hetzner IPs being banned from some registries. When I install Spegel via Helm, a daemonset is created and nodes exchange images as expected (confirmed both by the logs and by the pods starting on all nodes without any issues). But with the embedded registry mirror enabled with the flag I don't see any of this.

[brandond]

No spegel daemonset is created. I expected a daemonset to be created when adding --embedded-registry

Spegel is embedded in the main k3s process. What did you see that made you expect to see this deployed as a daemonset?

I will also note that you didn't mention configuring any registries in registries.yaml. Please reference the documentation again for required configuration.

[frederikhs]

No spegel daemonset is created. I expected a daemonset to be created when adding --embedded-registry

Spegel is embedded in the main k3s process. What did you see that made you expect to see this deployed as a daemonset?

I will also note that you didn't mention configuring any registries in registries.yaml. Please reference the documentation again for required configuration.

In the Architecture section of the Spegel readme, it is stated that

Spegel is deployed as a Daemonset on each node which acts as both the registry and mirror

And thus my understanding was that this was what was going to happen in k3s with the --embedded-registry flag.

I have added registries.yaml to the all nodes in a larger cluster and tried pull an image on one node, then deleting the image tag in the registry and scaling the deployment to see if the image is getting pulled using the embedded registry, and it worked! Is there a place i can look for logs so debugging later on?

[vitobotta]

Yeah I also thought there would be a daemonset because of the way standard Spegel works. So I am not sure why is it surprising that we had that kind of expectation somehow. I also got it working nicely in the meantime.

[brandond]

If you're using it standalone then the upstream docs are relevant. If you're using the embedded Spegel, I would stick to the k3s docs as it is literally embedded in and its configuration managed by k3s.