ACR doesn't support password grant in oauth workflow

[northtyphoon]

A couple of new container clients (eg, buildkit, img) based on containterd are sending password grand oauth request (username/pasword) to the registry auth server to the get access token.

Here is the containerd code that sends the oauth request.

https://github.com/containerd/containerd/blob/b28aa0e071df7b2ec6665a8d8926c9ffe20fa480/remotes/docker/resolver.go#L510

Currently, ACR doesn't support password grant in its oauth workflow and will fail the requests.

BTW: Docker CLI handles the basic auth slightly different so it can work with ACR. I opened an issue to confirm the behavior in containerd containerd/containerd#2381

/cc @yuwaMSFT2 @ehotinger

[yuwaMSFT2]

We should be able to support this additional mode in ACR's token server.

[cpressland]

We ran into this today while testing our migration from Docker to Containerd. Sorry for the stupid question, but, given that this seems to be a relatively simple fix what are the timeframes associated with a change like this hitting production?

[northtyphoon]

@cpressland , you can take a look at northtyphoon/img@2da858c which has a workaround for the auth issue in containerd (resolver.go). The commit also has another fix (pusher.go) and it has been merged to the containerd.

[cpressland]

Thanks @northtyphoon - I've just rebuilt containerd from source in a local docker container and still see the same result: ctr: failed to resolve reference "<image>": failed to fetch oauth token: unexpected status: 400 Bad Request while running: ./ctr image pull <image> -u <user>:<pass>. Any further advice for now?

[northtyphoon]

@cpressland I'm not sure. Can you double check if you indeed rebuilt with the change https://github.com/northtyphoon/img/blob/2da858cb88696e7a28384e3cbb9cc1d10d997bc8/vendor/github.com/containerd/containerd/remotes/docker/resolver.go#L524.
If it still doesn't work, can you send us (acrsup at microsoft dot com) your registry/image name?

[cpressland]

@northtyphoon it looks like resolver.go in the master branch doesn't have that change yet, however the version I built had the changes for pusher.go.

Once I patched in your change to my local resolver.go and recompiled I was able to pull just fine. Are you going to put in a pull request for https://github.com/containerd/containerd with the resolver.go change?

[northtyphoon]

@cpressland It's a hack and I don't plan to merge to contained. TBH I don't like the existing fallback code in resolver.go which is only targetting a few vendors.
We have started the work to fix it on ACR. May take a few weeks to roll out worldwide.

[northtyphoon]

We have rolled out the fix worldwide. Thanks @shizhMSFT @yuwaMSFT2

@cpressland if you build the latest img code, you should be able to pull/push to ACR.