Auth to Docker Hub registry not working

[laupow]

Description

We configured dockerd to authenticate every image pull request to Docker Hub to avoid rate limits. We also have private images in Docker Hub. Authenticating all image pull requests conveniently handles rate limit problems and private images.

As EKS clusters upgrade to 1.24 and beyond, we still want containerd to authenticate requests to Docker Hub to continue avoiding rate limits.

I have reviewed the Configure Registry Credentials docs but still do not have authenticated pulls to Docker Hub working with containerd.

Steps to reproduce the issue

1. Launch an EC2 instance amazon-eks-node-1.24-v20230513 into a 1.24 EKS cluster with self-managed nodes
2. Run this shell script after instance boots:

mkdir -p /etc/containerd/certs.d/docker.io
cat << EOF > /etc/containerd/certs.d/docker.io/hosts.toml
server = "https://docker.io"

[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
EOF

cat << EOF >> /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".auth]
username = "my-bot-username"
password = "my-bot-password"

EOF

sudo systemctl restart containerd

3. Run a crictl pull command:

/usr/local/bin/crictl -D -r unix:///run/containerd/containerd.sock  pull docker.io/my-company/my-app:8.3.0 
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/my-company/my-app:8.3.0,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
E0531 17:56:37.032746    3612 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"docker.io/my-company/my-app:8.3.0\": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:e882534f9e66c1cdd48a564587862cbbf106ed54333fe26a5e9a0595a0bd51bf: not found" image="docker.io/my-company/my-app:8.3.0"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/my-company/my-app:8.3.0": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:e882534f9e66c1cdd48a564587862cbbf106ed54333fe26a5e9a0595a0bd51bf: not found 

Describe the results you received and expected

What doesn't work

1. I expect crictl pull docker.io/my-company/my-app:8.3.0 to pull this private image successfully, using authentication

What does work:

1. Adding --username to crictl pull to validate credentials and network path

# /usr/local/bin/crictl -D -r unix:///run/containerd/containerd.sock pull --username my-bot-username docker.io/my-org/my-app:8.2.0 
DEBU[0000] get image connection                         
Enter Password:
DEBU[0001] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/my-org/my-app:8.2.0,Annotations:map[string]string{},},Auth:&AuthConfig{Username:my-bot-username,Password:my-bot-password,Auth:,ServerAddress:,IdentityToken:,RegistryToken:,},SandboxConfig:nil,} 
DEBU[0003] PullImageResponse: &PullImageResponse{ImageRef:sha256:69831caa6df16c470794c77e0e5ce8bed22835aaecb6dab87f0b1b3309537bbb,} 
Image is up to date for sha256:69831caa6df16c470794c77e0e5ce8bed22835aaecb6dab87f0b1b3309537bbb

2. Pulling public images:

 /usr/local/bin/crictl -D -r unix:///run/containerd/containerd.sock pull docker.io/node:18
DEBU[0000] get image connection                         
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/node:18,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,} 
DEBU[0008] PullImageResponse: &PullImageResponse{ImageRef:sha256:78b037dbb659142a12986b522897824ab60209d194a336c15a5006a55fffe9b0,} 
Image is up to date for sha256:78b037dbb659142a12986b522897824ab60209d194a336c15a5006a55fffe9b0

What version of containerd are you using?

containerd github.com/containerd/containerd 1.6.19 1e1ea6e

Any other relevant information

Versions

$ runc --version
runc version 1.1.4
commit: 5fd4c4d144137e991c4acebb2146ab1483a97925
spec: 1.0.2-dev
go: go1.18.6
libseccomp: 2.4.1

crictl version v1.26.0
$ /usr/local/bin/crictl -D -r unix:///run/containerd/containerd.sock  info 
### attached

crictl-info.json.txt

$ uname -a
Linux ip-192-168-1-87.ec2.internal 5.10.178-162.673.amzn2.x86_64 #1 SMP Mon Apr 24 23:34:06 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

/etc/containerd/config.toml

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5"

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".auth]
username = "my-bot-username"
password = "my-bot-password"

I have reviewed the following community discussions and there is no clear solution:
#8186
#5404
#6493

[laupow]

@dmcgowan I noticed you recently updated docs in #8186 (comment) (appreciate that!)

Is it possible to add a Docker Hub example to the registry credential docs? There is uncertainty on how to correctly make this configuration. I imagine many people may want to configure containerd to authenticate image pull requests to Docker Hub to avoid rate limits.

[israfilalam22]

Dear, I can't solve this issue: ctr image pull docker.io/user/image:tag
docker.io/israfilbat22/insight-dashboard-service:desco-v1: resolving |--------------------------------------|
elapsed: 1.9 s total: 0.0 B (0.0 B/s)
INFO[0002] trying next host error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/user/image:tag": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed....

here my config.toml file:

cat /etc/containerd/config.toml
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."registry-1.docker.io".auth]
username = "@@@@"
password = "######"

Note: When i user and password use in command line then ctr pull command working good but when i user and password put into config.toml then doesn't work.

ctr image pull --user 'user:password' docker.io/user/image:tag -----its working
ctr image pull docker.io/user/image:tag ---- its not working

i want to use " ctr image pull/push docker.io/user/image:tag " credential use from config.toml file

please suggest me

[dmcgowan]

@israfilalam22 this issue is related to using crictl pull which uses the CRI configuration. ctr image pull does not use this configuration. We don't recommend ctr as an end user tool, it's better to use docker or nerdctl and crictl for testing CRI configurations.