Registry TLS configuration from registries.yaml is only honored for mirror endpoints

[intrand]

Environmental Info:
K3s Version:

1.29.3+k3s1

Node(s) CPU architecture, OS, and Version:

Linux pi3 5.15.0-1049-raspi #52-Ubuntu SMP PREEMPT Thu Mar 14 08:39:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
1 controlplane node running etcd, 5 worker nodes, all matching raspberry pi computers

Describe the bug:
configuring /etc/rancher/k3s/registries.yaml with the bare minimum for a private registry with a self-signed cert no longer works, but downgrading to 1.29.2+k3s1 allows it to work again without any other changes.

---
configs:
  "registry.domain.tld":
    tls:
      ca_file: /usr/local/share/ca-certificates/ca_from_cluster.pem

Steps To Reproduce:

1. configure /etc/rancher/k3s/registries.yaml as above
2. install k3s using latest channel (currently version 1.29.3+k3s1)
3. deploy some container with its image from that registry
4. observe a certificate signed by unknown authority error emitted by containerd, captured in kubectl describe pod $pod_name events
5. downgrade to 1.29.2+k3s1
6. delete the pod
7. observe the image pulling without issue

Expected behavior:

to see the image pull correctly as it did in the previous release :)

Actual behavior:

errors related to tls verification and failed pulls

Additional context / logs:

not to lead you down a rabbit hole, but perhaps this is related? #9341

[brandond]

Can you confirm that you are not using a custom containerd config template? Can you provide the output of find /var/lib/rancher/k3s/agent/etc/containerd/ -type f -print -exec cat {} \; along with containerd.log showing the failed pull?

[intrand]

Can you confirm that you are not using a custom containerd config template? Can you provide the output of find /var/lib/rancher/k3s/agent/etc/containerd/ -type f -print -exec cat {} \; along with containerd.log showing the failed pull?

I have not touched the template at all. I also inspected the containerd toml and compared everything that seemed relevant to a backup from an earlier version and everything was identical.

I do not have the containerd log anymore. Are you unable to reproduce this behavior in 1.29.3+k3s1? 🤔 If absolutely need be I can destroy my cluster and build from scratch, but that should be the last resort.

EDIT: the cluster is up and running on 1.29.2+k3s1 with traffic going to/from. It's disruptive for me to test this on the same metal. I can try on another machine, but so can anyone :) it would be nice to see if anyone else can reproduce this

[brandond]

According to the containerd docs at https://github.com/containerd/containerd/blob/release/1.7/docs/hosts.md, all the host fields are valid at the root level:

For each registry host namespace directory in your registry config_path you may include a hosts.toml configuration file. The following root level toml fields apply to the registry host namespace:

This is what k3s generates:

root@systemd-node-1:/# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://172-17-0-7.sslip.io/v2"
capabilities = ["pull", "resolve", "push"]

ca = ["/usr/local/share/ca-certificates/registry.crt"]

However, containerd fails to load that:

time="2024-04-01T22:11:02.070675417Z" level=error msg="failed to decode hosts.toml" error="invalid `host` tree"

Apparently it goes looking for at least one host section; if it can't find one it fails to use the hosts.toml file entirely, despite the presence of valid config at the root level.

As a workaround, we can generate an empty host section; the following works properly:

root@systemd-node-1:/# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/hosts.toml
# File generated by k3s. DO NOT EDIT.

server = "https://172-17-0-7.sslip.io/v2"
capabilities = ["pull", "resolve", "push"]

ca = ["/usr/local/share/ca-certificates/registry.crt"]

[host]

I can address this in the next release. In the mean time, if you do not currently specify a port in your registry namespace, you should be able to work around the issue with something like this in your registries.yaml:

mirrors:
 172-17-0-7.sslip.io:
   endpoint:
     - https://172-17-0-7.sslip.io:443
configs:
 "172-17-0-7.sslip.io:443":
   tls:
     ca_file: /usr/local/share/ca-certificates/registry.crt

Note use of a port in the endpoint to force it to generate a host entry in the hosts.toml.

[brandond]

• Opened an upstream issue: Error parsing hosts.toml without any host tree containerd/containerd#10027
• And pull request: [release/1.7] Fix issue with empty host tree in hosts.toml containerd/containerd#10028

[intrand]

Thank you very much for going through the work to reproduce this, @brandond!

[brandond]

Using 172-17-0-7.sslip.io as an example registry, the two possible work-arounds are:

1. If your registry namespace does not currently include a port, configure a mirror endpoint with a port:

   mirrors:
     172-17-0-7.sslip.io:
       endpoint:
         - https://172-17-0-7.sslip.io:443
   configs:
     "172-17-0-7.sslip.io:443":
       tls:
         ca_file: /usr/local/share/ca-certificates/registry.crt

2. Manually drop the CA certificate into the registry namespace's configuration directory, and make it immutable so that k3s does not remove it when restarting:

   mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/
   cp /usr/local/share/ca-certificates/registry.crt /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/ca.crt
   chattr +i /var/lib/rancher/k3s/agent/etc/containerd/certs.d/172-17-0-7.sslip.io/ca.crt