No metrics on Spegel / Embedded Registry Mirror worker nodes

[reefland]

After reading https://docs.k3s.io/installation/registry-mirror

• The switch to enable embedded registry mirror is only supported on server nodes.
• When looking at Prometheus metrics, they are only provided for the servers nodes with switch enabled.

The docs state all nodes such as: When enabled at a cluster level, all nodes will host a local OCI registry on port 6443, and publish a list of available images via a peer to peer network on port 5001.

If all nodes are hosting, why are there only metrics for control-plane nodes with switch enabled?

Checking a Control-Plane node:

$ sudo netstat -ntlp | grep LISTEN | grep '6443\|5001'

tcp        0      0 192.168.10.215:5001     0.0.0.0:*               LISTEN      2998386/k3s server
tcp6       0      0 :::6443                 :::*                    LISTEN      2998386/k3s server

Checking a Worker node:

$ sudo netstat -ntlp | grep LISTEN | grep '6443\|5001'

tcp        0      0 192.168.10.218:5001     0.0.0.0:*               LISTEN      1946315/k3s agent
tcp6       0      0 :::6443                 :::*                    LISTEN      1946315/k3s agent

It does look look like all hosts have ports opened.

Is it just an issue with metrics not being exposed/collected on worker nodes?

[brandond]

This would appear to be an issue with your scraper config. The Spegel metrics are mixed in with the rest of the metrics, which are exposed via all metrics endpoints due to how K3s runs all the Kubernetes components in a single process. Check the metrics available on the kubelet metrics port, you should find them in there. That's where I usually poll my metrics from, since all nodes have a kubelet - while only server nodes have the apiserver + apiserver metrics endpoint.

See also: #9951

[reefland]

Thank you... that was the hint I needed.

$ k get --raw /api/v1/nodes/k3s04/proxy/metrics | grep "spegel"

libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 21
spegel_advertised_images{registry="ghcr.io"} 14
spegel_advertised_images{registry="quay.io"} 12
spegel_advertised_images{registry="registry.k8s.io"} 10
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 90
spegel_advertised_keys{registry="ghcr.io"} 211
spegel_advertised_keys{registry="quay.io"} 86
spegel_advertised_keys{registry="registry.k8s.io"} 109
# HELP spegel_mirror_requests_total Total number of mirror requests.
# TYPE spegel_mirror_requests_total counter
spegel_mirror_requests_total{cache="hit",registry="docker.io",source="internal"} 5
spegel_mirror_requests_total{cache="hit",registry="quay.io",source="internal"} 7
spegel_mirror_requests_total{cache="miss",registry="ghcr.io",source="internal"} 15

I did have a filter, adjusted and they show up now.