ErrImagePull when using latest docker image tag & embedded registry in air-gapped cluster

[matthewadams]

Environmental Info:
K3s Version:

k3s version v1.29.3+k3s1 (8aecc26b)
go version go1.21.8

Node(s) CPU architecture, OS, and Version:

Linux hp1 5.4.0-176-generic #196-Ubuntu SMP Fri Mar 22 16:46:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

Cluster Configuration:
3 servers, 0 agents

$ sudo cat /etc/systemd/system/k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
	'--cluster-init' \
	'--embedded-registry' \
	'--disable=traefik' \
	'--disable=servicelb' \
	'--tls-san=192.168.10.51,192.168.10.52,192.168.10.53' \
	'--node-ip=192.168.10.51' \
	'--node-external-ip=192.168.10.51' \
	'--flannel-iface=eno1' \

The file /etc/rancher/k3s/registries.yaml on all server nodes is:

mirrors:
  "*":

Describe the bug:
Given busybox.yaml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  terminationGracePeriodSeconds: 1
  containers:
    - name: busybox
      image: busybox # or, add :latest
      command:
        - sleep
      args:
        - "infinity"

and given that the referenced image is in the embedded registry:

$ sudo ctr i ls | grep busybox
docker.io/library/busybox:1.35.0                                                             application/vnd.oci.image.manifest.v1+json           sha256:4b9dd8bbde7eb5d60bc86ed64100692745310912e023e49fc85fd04ca82d7d10 4.3 MiB   linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/library/busybox:latest                                                             application/vnd.oci.image.manifest.v1+json           sha256:f19676f4de546985ff6d516bd5e772f2cacfae61699078c93cda3b4472371267 4.3 MiB   linux/amd64 io.cri-containerd.image=managed                                                             
docker.io/rancher/mirrored-library-busybox:1.36.1                                            application/vnd.docker.distribution.manifest.v2+json sha256:3ebd70b65321b7d390b8031ff280fbf59cd747e7a80490c9ee1bda13d6463e4e 4.3 MiB   linux/amd64 io.cattle.k3s.pinned=pinned,io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned 

then executing k3s kubectl apply -f busybox.yaml deploys the pod, but I get an ErrImagePull. Describing the pod in k9s shows:

Steps To Reproduce:

1. Create a k3s cluster with an embedded registry and the above /etc/rancher/k3s/registries.yaml file.
2. Copy the busybox.yaml content above.
3. Ensure busybox:latest is present in the embedded registry.
4. Apply with kubectl -f busybox.yaml

Expected behavior:
Pod should be able to pull image.

Actual behavior:
Pod unable to pull image, in ErrImagePull condition.

Additional context / logs:
If you omit the image tag or explicitly set tag to latest, this bug is present. However, if you use a tag other than latest, like 1.35.0, everything works ok.

[brandond]

This is covered in the documentation: https://docs.k3s.io/installation/registry-mirror#latest-tag

[matthewadams]

Oh wow. I must've missed that. I had no idea! Sorry for the noise.