/etc/rancher/k3s/registries.yaml is ignored

[zim32]

Environmental Info:
K3s Version:
k3s version v1.29.5+k3s1 (4e53a32)
go version go1.21.9

Node(s) CPU architecture, OS, and Version:
Linux 6.8.0-35-generic #35-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:51:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
1 server

Describe the bug:
I am trying to use private registry which is working over http. I am trying to add mirror so that proper endpoint is used but it does not change anything.

Steps To Reproduce:
Add this to /etc/rancher/k3s/registries.yaml:

mirrors:
  docker-registry.default.svc.cluster.local:
    endpoint:
      - "http://docker-registry.default.svc.cluster.local:5000"

Restart k3s:

sudo systemctl restart k3s

Try to pull image:

crictl pull docker-registry.default.svc.cluster.local:5000/symfony-api:1

Expected behavior:
Image is pulled

Actual behavior:
Got error

E0615 12:48:39.145915   20670 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker-registry.default.svc.cluster.local:5000/symfony-api:1\": failed to resolve reference \"docker-registry.default.svc.cluster.local:5000/symfony-api:1\": failed to do request: Head \"https://docker-registry.default.svc.cluster.local:5000/v2/symfony-api/manifests/1\": dial tcp: lookup docker-registry.default.svc.cluster.local: Try again" image="docker-registry.default.svc.cluster.local:5000/symfony-api:1"
FATA[0005] pulling image: failed to pull and unpack image "docker-registry.default.svc.cluster.local:5000/symfony-api:1": failed to resolve reference "docker-registry.default.svc.cluster.local:5000/symfony-api:1": failed to do request: Head "https://docker-registry.default.svc.cluster.local:5000/v2/symfony-api/manifests/1": dial tcp: lookup docker-registry.default.svc.cluster.local: Try again

[zim32]

The problem was because containerd is trying to resolve host docker-registry.default.svc.cluster.local from host machine and it fails. Issues solved.

[xingxing122]

How was this problem solved

[brandond]

By not expecting cluster DNS to be usable in containerd, which runs outside the cluster.