Fix remaining corner cases in containerd certs.d support

[brandond]

There are a couple remaining issues with the new containerd registries.yaml -> certs.d transition:

1. If the last registry in the endpoint list is the default, the registry overrides for that registry are not used - k3s will skip generating a certs.d/<registry>/hosts.toml for that registry and as a result containerd will not attempt to use the provided registry endpoints, it will always go out to the default.

mirrors:
  docker.io:
    endpoint:
      - registry.example.com
      - docker.io

This is kind of a strange thing to do anyway, so I doubt anyone is doing this.

2. Default TLS config from the wildcard entry is not honored:

configs:
  "*":
    tls:
      insecure_skip_verify: true

The docs do not cover use of the wildcard default registry for either mirrors or configs, and I don't know if it even worked for configs previously to begin with, and the upstream docs on this are not clear as to whether or not it is expected to work: https://github.com/containerd/containerd/blob/main/docs/hosts.md
UPDATE: It appears that containerd doesn't support loading tls config from the _default entry.

[ShylajaDevadiga]

1. If the last registry in the endpoint list is the default, the registry overrides for that registry are not used

Validated using k3s version v1.28.8-rc1+k3s1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:
Ubuntu 22.04

$ cat /etc/rancher/k3s/registries.yaml 
---
mirrors:
  docker.io:
    endpoint:
      - https://registry.example.com/
      - docker.io
  k8s.gcr.io:
    endpoint:
      - https://registry.example.com/
      - k8s.gcr.io
configs:
  registry.example.com:
    auth:
      username: testusername
      password: testpassword
    tls:
      ca_file: /home/ubuntu/ca.pem

Steps to validate

1. Configure private registry
2. Create the registries.yaml with the last registry in the endpoint list as the default to /etc/rancher/k3s
3. Install k3s.
4. Validate certs.d//hosts.toml is generated

Results from reproducing the issue:

ubuntu@ip-172-31-6-82:~$ k3s -v
k3s version v1.29.2+k3s1 (86f10213)
go version go1.21.7

$ sudo cat  /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
cat: /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml: No such file or directory

Results from validating the issue:

ubuntu@ip-172-31-12-26:~$ k3s -v
k3s version v1.29.3-rc1+k3s1 (8aecc26b)
go version go1.21.8

$ sudo cat  /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://k8s.gcr.io/v2"
capabilities = ["pull", "resolve", "push"]



[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

$ sudo cat  /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]



[host."https://registry.example.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

2. As Brad mentioned containerd doesn't support loading tls config from the _default entry

Passing CA cert or with insecure_skip_verify doesn’t support loading tls config from the _default/* entry.

ubuntu@ip-172-31-11-245:~$ cat /etc/rancher/k3s/registries.yaml 
---
mirrors:
  docker.io:
    endpoint:
      - registry.example.com
  k8s.gcr.io:
    endpoint:
      - registry.example.com
configs:
  "*":
    tls:
      insecure_skip_verify: true

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/_default/hosts.toml
# File generated by k3s. DO NOT EDIT.

skip_verify = true

Mar 22 17:18:04 ip-172-31-11-245 k3s[2055]: E0322 17:18:04.159884    2055 kuberuntime_manager.go:1171] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry.example.com/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority" pod="kube-system/helm-install-traefik-crd-zrvm7"