Containerd fails to authenticate to a private Google Cloud Registry

[j0nd0n7]

What I'd like to achieve is to use a private Google Cloud Registry to pull images from microk8s.

To configure the registry, I opted to modify the containerd-template.toml file instead of using ImagePullSecrets.

These is what I tried:
First I created a keyfile.json from the Google Cloud Console like explained here:

The result is a text file with a JSON value (the long term credentials).

Second (These are my different tries)
try 1) Edit the /var/snap/microk8s/current/args/containerd-template.toml

[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
     [plugins.cri.registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io"]
     [plugins.cri.registry.mirrors."gcr.io"]
        endpoint = ["https://gcr.io"]

[plugins.cri.registry.configs."gcr.io".auth]
  username = "_json_key"
  password = 'the json content without new lines'

Note _json_key is a especial username GCP uses to authenticate through json keyfile.

Then restart microk8s microk8s.stop && microk8s.start

When I check if microk8s.ctr image pull gcr.io/my_project_id/my_image:tag works this is the output

ctr: failed to dial "/var/snap/microk8s/common/run/containerd.sock": context deadline exceeded

try 2)
As I read here that plugins.cri.registry.configs."gcr.io".auth fields have the same meaning that the .docker/config.json fields, I logged in to the private registry cat keyfile.json | docker login -u _json_key --password-stdin https://gcr.io (to generate the entries in .docker/config.json)

That added these content to the config:

{
	"auths": {
		"gcr.io": {
			"auth": "a_huge_token"
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/18.09.7 (linux)"
	}
}

Then I take the auth token value and replaced the username and password entries with an auth entry in the file /var/snap/microk8s/current/args/containerd-template.toml like this:

[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
     [plugins.cri.registry.mirrors."docker.io"]
        endpoint = ["https://registry-1.docker.io"]
     [plugins.cri.registry.mirrors."gcr.io"]
        endpoint = ["https://gcr.io"]

[plugins.cri.registry.configs."gcr.io".auth]
  auth = "a_huge_token"

Then restart microk8s microk8s.stop && microk8s.start

When I check if microk8s.ctr image pull gcr.io/my_project_id/my_image:tag works this is the output

ctr: failed to resolve reference "gcr.io/my_project_id/my_image:tag": unexpected status code https://gcr.io/v2/my_project_id/my_image/manifests/tag: 401 Unauthorized

I really appreciate some directions to address this registry config.
Thank you

[j0nd0n7]

I also tried using

[plugins.cri.registry.auths]
      [plugins.cri.registry.auths."gcr.io"]

instead of

[plugins.cri.registry.configs."gcr.io".auth]

The output is

ctr: failed to resolve reference "gcr.io/my_project_id/my_image:tag": unexpected status code https://gcr.io/v2/my_project_id/my_image/manifests/tag: 401 Unauthorized

[balchua]

Can you try using this?

[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://gcr.io"]
    username = ""
    password = ""
    auth = ""
    identitytoken = ""

Take note of the "https" at the config. I think it points to the endpoint.

[j0nd0n7]

Unfortunately, that didn't solve the issue, although you are right that the endpoint must be used there.

The po describe prints:

Failed to pull image "gcr.io/my_project_id/my_image:tag": rpc error: code = Unknown desc = failed to resolve image "gcr.io/my_project_id/my_image:tag": no available registry endpoint: invalid auth config

I also tried the https change including the credential instead of having it empty.
In this case the po describe prints:

Failed to pull image "gcr.io/my_project_id/my_image:tag": rpc error: code = Unknown desc = failed to resolve image "gcr.io/my_project_id/my_image:tag": no available registry endpoint: unexpected status code https://gcr.io/v2/my_project_id/my_image/manifests/tag: 403 Forbidden

In both cases, the microk8s.ctr image pull gcr.io/my_project_id/my_image:tag output is:

ctr: failed to resolve reference "gcr.io/my_project_id/my_image:tag": unexpected status code https://gcr.io/v2/my_project_id/my_image/manifests/tag: 401 Unauthorized

Note the 401 error instead the 403. It seams microk8s.ctr image pull uses a different config than the pod image pull.

[j0nd0n7]

Finally it's working (a day and a half fighting).
What was wrong:

• First: what @balchua pointed about using the endpoint including the https:// (thank you!)
• Second: Miss chosen the GCP Role (Now I'm using Storage Admin. For pull only Storage Viewer should work too)

This is how the config looks:

[plugins.cri.registry]
  [plugins.cri.registry.mirrors]
    [plugins.cri.registry.mirrors."docker.io"]
      endpoint = ["https://registry-1.docker.io"]
    [plugins.cri.registry.mirrors."gcr.io"]
      endpoint = ["https://gcr.io"]
[plugins.cri.registry.auths]
  [plugins.cri.registry.auths."https://gcr.io"]
    username = "_json_key"
    password = "{\n  \"type\": \"service_account\",\n  \"project_id\": ...
    email = "an@email.com"
    auth = "X2pz...."

Where:

• username: must be "_json_key"
• password: is the keyfile.json content with quotes end new lines scaped
• email: is any well formatted email
• auth: is the base64 encoded of _json_key:the_keyfile_content

Maybe there are some redundant fields. I'll check it out...

[balchua]

Thank you @j0nd0n7 for the detailed steps!

[DazWilkin]

I'm unable to get this to work and it's driving me nuts :-)

@j0nd0n7 solution mostly makes sense but, if replicate it, I receive 401s from GCR.

It should not be necessary to duplicate username|password and auth; either username|password or auth should be required since auth = f(username,password)

I would also prefer to use an access token rather than a service account.

Please see the thread referenced by @mikebrow above: containerd/cri#1482

The plugins.cri.registry config is slightly confusing and appears to be undergoing change. It does not appear that containerd balks at bad configurations. I can, for example r/auths/bauths and this doesn't (appear to) generate a bunch of bad configuration errors. So, it's also unclear to me whether I'm even using the correct configuration.