[Release-1.28] - Registry TLS configuration from registries.yaml is only honored for mirror endpoints

[brandond]

Backport fix for Registry TLS configuration from registries.yaml is only honored for mirror endpoints

• Registry TLS configuration from registries.yaml is only honored for mirror endpoints #9839

[aganesh-suse]

Validated on release-1.28 branch with version v1.28.9-rc1+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

registries.yaml:

 $ sudo cat /etc/rancher/k3s/registries.yaml
mirrors:
  pvt-registry.com:
    endpoint:
      - pvt-registry.com
  docker.io:
    endpoint:
      - pvt-registry.com      
  k8s.gcr.io:
    endpoint:
      - pvt-registry.com      
configs:
  pvt-registry.com:
    auth:
      username: xxxx
      password: xxxx
    tls:
      ca_file: /home/user/ca.pem

test-image.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: pvt-reg-test
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/warn: privileged
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pvt-reg-test
  namespace: pvt-reg-test
spec:
  selector:
    matchLabels:
      k8s-app: nginx-app-clusterip
  replicas: 2
  template:
    metadata:
      labels:
        k8s-app: nginx-app-clusterip
    spec:
      containers:
      - name: nginx
        image: pvt-registry.com/nginx:latest
        ports:
        - containerPort: 8080

Testing Steps

1. Copy config.yaml and registries.yaml

$ sudo mkdir -p /etc/rancher/k3s 
$ sudo cp config.yaml /etc/rancher/k3s
$ sudo cp registries.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.28.9-rc1+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Push an image onto the private registry and try to deploy a pod with said image.
   The image should get pulled and pod should come up without any tls certificate errors.

$ kubectl apply -f test-image.yaml
$ kubectl get pods -n pvt-reg-test
$ kubectl describe pod/pvt-reg-test-abcd -n pvt-reg-test

5. Check the hosts.toml files for host section

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.28.8+k3s1 (653dd61a)
go version go1.21.8

$ kubectl get pods -A
kube-system      coredns-6799fbcd5-2pwk7                   1/1     Running            0          4m57s
kube-system      helm-install-traefik-8fdr8                0/1     Completed          1          4m57s
kube-system      helm-install-traefik-crd-htrst            0/1     Completed          0          4m57s
kube-system      local-path-provisioner-6c86858495-8tp6b   1/1     Running            0          4m57s
kube-system      metrics-server-54fd9b65b-k7xx9            1/1     Running            0          4m57s
kube-system      svclb-traefik-8f7984b5-2sgsq              2/2     Running            0          4m42s
kube-system      svclb-traefik-8f7984b5-9nvcn              2/2     Running            0          2m38s
kube-system      svclb-traefik-8f7984b5-pwvfr              2/2     Running            0          4m41s
kube-system      svclb-traefik-8f7984b5-rzgjj              2/2     Running            0          3m26s
kube-system      traefik-f4564c4f4-wclsq                   1/1     Running            0          4m42s
pvt-reg-test     pvt-reg-test-7494d967dd-9clx2             0/1     ErrImagePull       0          28s
pvt-reg-test     pvt-reg-test-7494d967dd-scbf7             0/1     ErrImagePull       0          28s

Pod Events:

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  6m28s                  default-scheduler  Successfully assigned pvt-reg-test/pvt-reg-test-7494d967dd-9clx2 to ip-172-31-27-81
  Normal   Pulling    5m7s (x4 over 6m28s)   kubelet            Pulling image "pvt-registry.com/nginx:latest"
  Warning  Failed     5m7s (x4 over 6m28s)   kubelet            Failed to pull image "pvt-registry.com/nginx:latest": failed to pull and unpack image "pvt-registry.com/nginx:latest": failed to resolve reference "pvt-registry.com/nginx:latest": failed to do request: Head "https://pvt-registry.com/v2/nginx/manifests/latest": tls: failed to verify certificate: x509: certificate signed by unknown authority
  Warning  Failed     5m7s (x4 over 6m28s)   kubelet            Error: ErrImagePull
  Warning  Failed     4m44s (x6 over 6m28s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    77s (x21 over 6m28s)   kubelet            Back-off pulling image "pvt-registry.com/nginx:latest"

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.28.9-rc1+k3s1 (b33e710f)
go version go1.21.9

$ kubectl get pods -A
kube-system      coredns-6799fbcd5-cswlf                   1/1     Running     0          4m58s
kube-system      helm-install-traefik-crd-6wqpn            0/1     Completed   0          4m59s
kube-system      helm-install-traefik-dw9r7                0/1     Completed   1          4m59s
kube-system      local-path-provisioner-6c86858495-75t9h   1/1     Running     0          4m58s
kube-system      metrics-server-54fd9b65b-2dknk            1/1     Running     0          4m58s
kube-system      svclb-traefik-9f46cb91-8d7fb              2/2     Running     0          3m28s
kube-system      svclb-traefik-9f46cb91-kc8ps              2/2     Running     0          4m41s
kube-system      svclb-traefik-9f46cb91-q759g              2/2     Running     0          4m41s
kube-system      svclb-traefik-9f46cb91-x2qxk              2/2     Running     0          2m38s
kube-system      traefik-7d5f6474df-cjhmq                  1/1     Running     0          4m41s
pvt-reg-test     pvt-reg-test-6567d988b5-9krx6             1/1     Running     0          29s
pvt-reg-test     pvt-reg-test-6567d988b5-x8czt             1/1     Running     0          29s

Check hosts.toml file contents:

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/pvt-registry.com/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://pvt-registry.com/v2"
capabilities = ["pull", "resolve", "push"]

ca = ["/home/ubuntu/ca.pem"]


[host]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://pvt-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.

server = "https://k8s.gcr.io/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://pvt-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]