Mirrors configured in /etc/rancher/k3s/registries.yaml do not take effect

[kingsd041]

Environmental Info:
K3s Version: v1.28.7+k3s1

Node(s) CPU architecture, OS, and Version: ubuntu 22.04

Cluster Configuration: 1 server

Describe the bug:

When configuring mirrors in /etc/rancher/k3s/registries.yaml, the mirrors do not take effect after starting K3s.

Steps To Reproduce:

• Configure /etc/rancher/k3s/registries.yaml with the desired mirrors.

mkdir -p /etc/rancher/k3s/

cat >> /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  "docker.io":
    endpoint:
      - "https://docker.nju.edu.cn/"
      - "https://registry-1.docker.io"
EOF

• Install K3s using the installation script (curl -sfL https://get.k3s.io | sh -).
• Wait for K3s to start successfully.
• Check the configured mirrors using crictl info.

    "registry": {
      "configPath": "/var/lib/rancher/k3s/agent/etc/containerd/certs.d",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
    },

Actual behavior:

The configured mirrors are not applied, and the mirrors field remains null in the output of crictl info.

Additional context / logs:

[brandond]

I believe this is a duplicate of #9341

Remove the second endpoint, you don't need to specify the default endpoint; it is always tried last - and on this specific release it triggers a bug in the registries configuration.

[kingsd041]

@brandond I removed the second default endpoint, but the situation is still the same.

root@ip-172-31-15-10:/etc/rancher/k3s# cat registries.yaml
mirrors:
  "docker.io":
    endpoint:
      - "https://docker.nju.edu.cn/"
root@ip-172-31-15-10:/etc/rancher/k3s#
root@ip-172-31-15-10:/etc/rancher/k3s# systemctl restart k3s
root@ip-172-31-15-10:/etc/rancher/k3s# crictl info | grep -A 5 "registry"
    "registry": {
      "configPath": "/var/lib/rancher/k3s/agent/etc/containerd/certs.d",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null

[brandond]

I'm not sure that crictl info will show mirrors when configPath is in use; you have to actually look at the filesystem. Have you checked to confirm that /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml exists and has the correct content?

ref: https://github.com/containerd/containerd/blob/main/docs/hosts.md

[kingsd041]

/var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml of v1.28.7+k3s1 shows the mirror I configured：

root@ip-172-31-4-143:~# cat /etc/rancher/k3s/registries.yaml
mirrors:
  "docker.io":
    endpoint:
      - "https://docker.nju.edu.cn/"

root@ip-172-31-4-143:~# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"

[host."https://docker.nju.edu.cn/v2"]
  capabilities = ["pull", "resolve"]

However, I also tried the same operation using v1.28.5+k3s1. Using v1.28.5+k3s1, I can display the mirror information through crictl info:

root@ip-172-31-10-120:~# k3s -v
k3s version v1.28.5+k3s1 (5b2d1271)
go version go1.20.12

root@ip-172-31-10-120:~# crictl info | grep -A 5 "registry"
    "registry": {
      "configPath": "",
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://docker.nju.edu.cn/"

root@ip-172-31-10-120:~# cat /etc/rancher/k3s/registries.yaml
mirrors:
  "docker.io":
    endpoint:
      - "https://docker.nju.edu.cn/"

Moreover, I also observed that the Containerd versions of v1.28.7+k3s1 and v1.28.5+k3s1 are both v1.7.11-k3s2, @brandond Do you know the reason why v1.28.7+k3s1 cannot display mirror through crictl info?

[brandond]

Because we switched to using hosts.d (configPath) instead of the deprecated inline mirrors config.

• Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972
• Add support for containerd cri registry config_path #8973