Different pull behavior for short-name images (gke-nvidia-installer:fixed) between containerd and CRI-O

[beratturan]

Description:
When pulling an image with an unqualified short name, containerd and CRI-O behave differently.

On a GKE node with containerd, the pull of gke-nvidia-installer:fixed succeeds (resolved as docker.io/library/gke-nvidia-installer:fixed via mirrors).

On a GKE node with CRI-O + sysbox, with equivalent registry mirror config (mirror.gcr.io + registry-1.docker.io), the same pull fails with:

rpc error: code = Unknown desc = initializing source docker://gke-nvidia-installer:fixed:
(Mirrors also failed: 
 [mirror.gcr.io/library/gke-nvidia-installer:fixed: manifest unknown]
 [registry-1.docker.io/library/gke-nvidia-installer:fixed: requested access to the resource is denied])

Environment:
• GKE nodes with GPU (V100 / sysbox runtime class) (v1.32.2-gke.200)
• One node running containerd (working) (github.com/containerd/containerd 1.7.24-0ubuntu024.04.1gke1)
• One node running CRI-O (failing)
• CRI-O version: 1.32.4
• Containerd version: <fill in output of containerd --version>
• Configured /etc/containers/registries.conf.d/01-docker-io-mirrors.conf to mirror docker.io via mirror.gcr.io and registry-1.docker.io.

Steps to reproduce:

1. Configure mirrors for docker.io in CRI-O:

[[registry]]
location = "docker.io"
prefix   = "docker.io"

  [[registry.mirror]]
  location = "mirror.gcr.io"

  [[registry.mirror]]
  location = "registry-1.docker.io"

2. Restart CRI-O.
sudo crictl -r /var/run/crio/crio.sock pull gke-nvidia-installer:fixed

Notes:
• No imagePullSecrets or Docker credentials are present on either node.
• Kubelet credentials are looking same in both node. (--image-credential-provider-config=/etc/srv/kubernetes/cri_auth_config.yaml)

[ctalledo]

Hi @beratturan, thanks for giving Sysbox a try.

The problem is not related to Sysbox per-se, since it has nothing to do with image pulling (that's purely a containerd or CRI-O operation).

Having said that, based on this error:

 [registry-1.docker.io/library/gke-nvidia-installer:fixed: requested access to the resource is denied])

That appears to be an authentication / credentials problem, meaning CRI-O does not have the credentials to pull the registry-1.docker.io/library/gke-nvidia-installer:fixed image.

I am not very familiar with how K8s manages registry credentials, so can't help further, but I have a few suggestions:

1. Check the CRI-O logs on the K8s node (sudo journalctl -u crio); the logs may provide more detail about the image pull error.
2. Verify that K8s (e.g., Kubelet) is in fact passing the registry credential info to CRI-O (not quite sure how it does that).
3. Check the CRI-O repo on GitHub to see of any similar issues.

Note: the CRI-O fork that Sysbox uses is 99% identical to upstream CRI-O, it only carries a simple change to ensure containers start with appropriate user-namespace ID mappings (i.e., the changes is not related to image pull).

Hope that helps!