ctr does not read /etc/containerd/config.toml credential settings when pulling images from ghcr.io

[effy-coding]

Description

I added ghcr.io private registry auth config to /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.configs]
    [plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io".auth]
      username = "UESRNAME"
      password = "CR_PAT"

And restart containerd

systemctl restart containerd

And check containerd config settings whether it was updated

(printing my ghcr.io settings)

containerd config dump

[plugins."io.containerd.grpc.v1.cri".registry.configs]

        [plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io"]

          [plugins."io.containerd.grpc.v1.cri".registry.configs."ghcr.io".auth]
            auth = ""
            identitytoken = "="
            password = "USERNAME"
            username = "CR_PAT"

Steps to reproduce the issue

No response

Describe the results you received and expected

Problem is here when i pull some images from my github organization package registry, ghcr.io returns 401 error

bash-5.0# ctr --namespace k8s.io --debug --address "/run/dockershim.sock" i pull ghcr.io/some-image                
DEBU[0000] fetching                                      image="ghcr.io/some-image"
DEBU[0000] resolving                                     host=ghcr.io
DEBU[0000] do request                                    host=ghcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+bottlerocket request.method=HEAD url="https://ghcr.io/v2/some-image/manifests/base"
DEBU[0000] fetch response received                       host=ghcr.io response.header.content-length=73 response.header.content-type=application/json response.header.date="Thu, 10 Feb 2022 06:31:15 GMT" response.header.www-authenticate="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:some-image:pull\"" response.header.x-github-request-id="A0B1:22FB:7DB6:17A96:6204B133" response.status="401 Unauthorized" url="https://ghcr.io/v2/some-image/manifests/base"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:some-image:pull\"" host=ghcr.io
DEBU[0000] do request                                    host=ghcr.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+bottlerocket request.method=HEAD url="https://ghcr.io/v2/some-image/manifests/base"
INFO[0000] trying next host                              error="failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized" host=ghcr.io
ctr: failed to resolve reference "ghcr.io/some-image": failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized

I think Authorization: Bearer $(echo $CR_PAT | base64) should be included but it wasn't

Is this bug?

What version of containerd are you using?

containerd github.com/containerd/containerd 1.5.9+bottlerocket 1407cab

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[AkihiroSuda]

ctr does not use CRI API, so it does not read any configuration from [plugins."io.containerd.grpc.v1.cri"].

If you want to debug the CRI config, you have to use crictl.

If you aren't interested in CRI and you just want to run a containerd container with GHCR, you can just use ctr --user USER:PASS:

containerd/cmd/ctr/commands/commands.go

Line 64 in 0340e84

Usage: "user[:password] Registry user and password",

See also nerdctl: https://github.com/containerd/nerdctl/blob/master/docs/registry.md#github-container-registry-ghcr

[effy-coding]

Thanks @AkihiroSuda!

But i am debugging kubelet's image pulling step (it fails), not ctr or containerd. So putting user:pass shoudn't be solution isn't it?

[effy-coding]

kubelet uses same container runtime endpoint with ctr (--container-runtime-endpoint=unix:///run/dockershim.sock). That's why i tried to debug this problem using ctr

/usr/bin/kubelet --cloud-provider aws --kubeconfig /etc/kubernetes/kubelet/kubeconfig --config /etc/kubernetes/kubelet/config --container-runtime=remote --container-runtime-endpoint=unix:///run/dockershim.sock --containerd=/run/dockershim.sock --network-plugin cni --root-dir /var/lib/kubelet --cert-dir /var/lib/kubelet/pki ...

[AkihiroSuda]

But i am debugging kubelet's image pulling step (it fails), not ctr or containerd.

Please try crictl, not ctr in this case: https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

[effy-coding]

Thanks @AkihiroSuda

But Bottlerocket OS seems using ctr, not crictl

Can i configure private registry auth of ctr? How can i find documentation?

[AkihiroSuda]

Can i configure private registry auth of ctr?

ctr --user USER:PASS

How can i find documentation?

ctr is undocumented because it is just a debugging tool and is not intended to be used by end users.

[Alceatraz]

ctr i pull --user docker:docker /:****

That solve me problem too.

[adamsb6]

I just spent half a day trying to figure out why the docs on how to configure a registry mirror for containerd apparently weren't working. I don't think most would expect that this CLI and Kubernetes would be so divergent. I expected that I could use ctr i pull to verify that all image pulls would be working as expected.

What's the reasoning for having ctr pull images in this way?

[AkihiroSuda]

What's the reasoning for having ctr pull images in this way?

Because ctr is not an CRI application.

In containerd v2.0, we are likely to have the common credential API that will work for both ctr and CRI. (Details are still undecided as of the time of posting this comment)

• [transfer] credential manager plugin #8228