[Release-1.28] - Image pulls from embedded registry fail if --bind-address is set (and not 127.0.0.1)

[brandond]

Backport fix for Image pulls from embedded registry fail if --bind-address is set (and not 127.0.0.1)

• Image pulls from embedded registry fail if --bind-address is set (and not 127.0.0.1) #10094

[aganesh-suse]

Validated on release-1.28 branch with version v1.28.11-rc2+k3s1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
embedded-registry: true
bind-address: x.x.x.x

registry.yaml:

mirrors:
  private.registry.com:
    endpoint:
      - private.registry.com
  docker.io:
    endpoint:
      - private.registry.com      
  k8s.gcr.io:
    endpoint:
      - private.registry.com      
configs:
  private.registry.com:
    auth:
      username: <username>
      password: <password>
    tls:
      ca_file: /home/ubuntu/ca.pem

Testing Steps

1. Copy config.yaml and registries.yaml:

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s && sudo cp registries.yaml /etc/rancher/k3s

P.S: Remember to copy over the /home/ubuntu/ca.pem for the registry settings to work

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.28.11-rc2+k3s1' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

Replication Results:

• k3s version used for replication:

$ k3s -v
k3s version v1.28.10+k3s1 (a4c5612e)
go version go1.21.9

$ journalctl -xeu k3s-agent | grep 'received image event' 

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.28.11-rc2+k3s1 (dd86a058)
go version go1.21.11

Sample spegel logs from the journal logs on the agent node:

$ journalctl -xeu k3s-agent | grep 'received image event'  
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause:3.6@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:51 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:51Z" level=info msg="spegel 2024/06/14 17:50:51 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"docker.io/rancher/mirrored-pause@sha256:74bf6fc6be13c4ec53a86a5acf9fdbc6787b176db0693659ad6ac89f115e182c\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer:unprivileged@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:50:55 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:50:55Z" level=info msg="spegel 2024/06/14 17:50:55 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/mytestcontainer@sha256:7e418465981575a9abef4ee16a80c562a2d2d171e591c1475c38347ef3ec2a72\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx:latest@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""
Jun 14 17:51:29 ip-172-31-24-105 k3s[208743]: time="2024-06-14T17:51:29Z" level=info msg="spegel 2024/06/14 17:51:29 \"level\"=0 \"msg\"=\"received image event\" \"image\"=\"private.registry.com/nginx@sha256:80550935209dd7f6b2d7e8401b9365837e3edd4b047f5a1a7d393e9f04d34498\" \"type\"=\"CREATE\""