RegistryToken is not refreshed on the re-use of docker resolver

[shizhMSFT]

Description

The registry token sent to the registry is not automatically refreshed even if the credentials are provided.

Taking docker hub as an example, the access token / registry token for anonymous pull is around 300s (i.e. 5 mins). If a reference is resolved and then resolve it again after 5 mins, the docker resolver will report an error. Docker resolver should retry by itself and should not rely on the call to retry.

Steps to reproduce the issue

No response

Describe the results you received and expected

Running the following code:

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"

	"github.com/containerd/containerd/remotes/docker"
)

func main() {
	ref := "docker.io/library/hello-world:latest"
	ctx := context.Background()
	resolver := docker.NewResolver(docker.ResolverOptions{})

	_, _, err := resolver.Resolve(ctx, ref)
	fmt.Println(err)

	time.Sleep(6 * time.Minute)

	_, _, err = resolver.Resolve(ctx, ref)
	fmt.Println(err)
}

Actual output:

<nil>
time="2021-12-14T22:56:15+08:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: invalid_token: authorization failed" host=registry-1.docker.io
pull access denied, repository does not exist or may require authorization: server message: invalid_token: authorization failed

Expected output:

<nil>
<nil>

What version of containerd are you using?

v1.6.0-beta.3

Any other relevant information

No response

Show configuration if it is related to CRI plugin.

No response

[bcspragu]

Reviving this thread, I'm hitting this now, on the most recent version (1.6.16), with a configuration like:

dkr := docker.NewResolver(docker.ResolverOptions{
	Hosts: docker.ConfigureDefaultRegistries(
		docker.WithAuthorizer(
			docker.NewDockerAuthorizer(
				docker.WithAuthCreds(func(in string) (string, string, error) {
						return someUsername, somePassword, nil
				}),
			),
		),
		docker.WithPlainHTTP(func(in string) (bool, error) {
			// For local testing
			return strings.Contains(in, "localhost"), nil
		}),
	),
})

After a few minutes, I get an error like:

pull access denied, repository does not exist or may require authorization: server message: invalid_token: authorization failed

I'm using a private but mostly off-the-shelf private Docker Registry, with token auth.

Is this a bug, or am I just holding it wrong? If the latter, what's the correct way to init the resolver or refresh the token?

[bcspragu]

One workaround I just tested is to create a new resolver (i.e.with docker.NewResolver) every time you want to pull an image. This is likely inefficient (because there's no opportunity to use cached credentials) and will be prohibitively expensive in an environment where images are constantly being pulled, but it's a fine crutch for my use case for now.