ACR Datasource unauthorized while fetching tags list. #14708

vlagorce · 2022-03-17T11:11:33Z

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

32.6.0

Please select which platform you are using if self-hosting.

GitLab self-hosted

If you're self-hosting Renovate, tell us what version of the platform you run.

14.8.2

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

Using ACR. Fetching new docker image tags is failing.

DEBUG: Datasource unauthorized (repository=MY/repository)
       "datasource": "docker",
       "packageName": "redacted.azurecr.io/upstream/rust",
       "url": "https://redacted.azurecr.io/v2/upstream/rust/tags/list?n=10000"

I tried to reproduced the call and authentication performed by renovate.
I'm facing the same authentication issue.

{
    "errors": [
        {
            "code": "UNAUTHORIZED",
            "message": "authentication required, visit https://aka.ms/acr/authorization for more information.",
            "detail": [
                {
                    "Type": "repository",
                    "Name": "upstream/rust",
                    "Action": "metadata_read"
                }
            ]
        }
    ]
}

Renovate is requesting a token with the following scope. repository:upstream/rust:pull.

Scope should be either repository:upstream/rust:pull,metadata_read or repository:upstream/rust:*

Relevant debug logs

Logs

Copy/paste any log here, between the starting and ending backticks

Have you created a minimal reproduction repository?

No reproduction repository

The text was updated successfully, but these errors were encountered:

viceice · 2022-03-17T11:14:53Z

Is metadata_read scope an official docker api scope or is that ACR specific? Does the normal docker client work as expected?

vlagorce · 2022-03-17T11:43:06Z

Seems to be ACR specific.
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions#authenticate-with-token (metadata/read)

viceice · 2022-03-18T08:32:09Z

@rarkins This PR is only a workaround which can break any time soon. A better solution is to make a HEAD request to the real resource (like tags list) and use the auth header values to get a token, looks like this is the way other docker api clients work. WDYT?

rarkins · 2022-03-18T08:56:40Z

Sounds good

vlagorce · 2022-03-21T10:49:04Z

make a HEAD request to the real resource (like tags list) and use the auth header values to get a token
@rarkins @viceice
Doing a head request should be

for ACR when doing getDockerApiTags
for any registry when doing getDockerApiTags
for any registry when performing getAuthHeaders

I start to implement something but the amount of test to fix could be big

viceice · 2022-03-21T11:38:08Z

@vlagorce What's the scope, when doing a head on /v2 endpoint? PR looks promissing.

@rarkins I'm not sure if we cache the token response, so it it maybe miss scopes for fetching the json manifests?

vlagorce · 2022-03-21T12:53:25Z

curl --location --head 'https://12345678.azurecr.io/v2/'
HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://12345678.azurecr.io/oauth2/token",service="12345678.azurecr.io"
...

curl --location --head 'https://12345678.azurecr.io/v2/my/node/tags/list'
HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://12345678.azurecr.io/oauth2/token",service="12345678.azurecr.io",scope="repository:my/node:metadata_read"
...

curl --location --head 'https://12345678.azurecr.io/v2/my/node/manifests/latest' 
HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://12345678.azurecr.io/oauth2/token",service="12345678.azurecr.io",scope="repository:my/node:pull"
...

vlagorce · 2022-03-21T13:14:26Z

In my case the token is valid 10min. I do not know if it is set by our admin or the default settings.

There is an other option if we don't call first the resource with HEAD.
When the authentication is rejected for a wrong scope the header contains a error message with insufficient_scope.
In that case the expected scope is provided too.

curl -i --location --request GET 'https://12345678.azurecr.io/v2/my/node/tags/list' \
--header 'Authorization: Bearer token-pull-instead-of-metadata_read'
HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://12345678.azurecr.io/oauth2/token",service="12345678.azurecr.io",scope="repository:my/node:metadata_read",error="insufficient_scope"
...

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required, visit https://aka.ms/acr/authorization for more information.","detail":[{"Type":"repository","Name":"my/node","Action":"metadata_read"}]}]}

viceice · 2022-04-08T11:52:45Z

@vlagorce What does curl --location --get 'https://12345678.azurecr.io/v2/' returns as auth header?

viceice · 2022-04-08T11:53:52Z

@rarkins It seems the docker daemon does a GET request, so maybe we need to do this too instead of the HEAD request🤔

vlagorce · 2022-04-08T12:13:56Z

Current issue is the need of getting the expected scope action to use the endpoint tag/list.
Which is not performed by the daemon if I'm not mistaken.

docker daemon/cli only pull or push images/tags

curl -iL --get https://123456.azurecr.io/v2/
HTTP/1.1 401 Unauthorized
Server: openresty
Date: Fri, 08 Apr 2022 12:12:17 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 149
Connection: keep-alive
Access-Control-Expose-Headers: Docker-Content-Digest
Access-Control-Expose-Headers: WWW-Authenticate
Access-Control-Expose-Headers: Link
Access-Control-Expose-Headers: X-Ms-Correlation-Request-Id
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
Www-Authenticate: Bearer realm="https://123456.azurecr.io/oauth2/token",service="123456.azurecr.io"
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 12307f6a-249c-4c8d-bbc4-13409af3d711
Strict-Transport-Security: max-age=31536000; includeSubDomains

Scope is missing.

viceice · 2022-04-08T12:16:37Z

ok, so the daemon doesn't need metadata_read, because it's only required for the tags list? 🤔

vlagorce · 2022-04-08T12:47:49Z

ok, so the daemon doesn't need metadata_read, because it's only required for the tags list?

It is a guess. I do not have a complete knowledge about what it could achieve. Looking at the doc and the code I don't see any other scope action than pull, push, mount, delete. It seems they are always requesting them both.

some doc https://github.com/distribution/distribution/blob/main/docs/spec/auth/token.md (that the lib use by docker )
https://github.com/distribution/distribution/blob/main/docs/spec/auth/scope.md

vlagorce · 2022-04-11T07:36:44Z

To summarize.
Azure Container Regitry (ACR) use a "custom" actionmetadata_read for the repo/tags/list.
When not authenticated HEAD and GET methods return

Bearer realm="https://12345678.azurecr.io/oauth2/token",service="12345678.azurecr.io",scope="repository:my-registry/my-image:metadata_read"

Other container registry (CR) might not support HEAD. Other CR expect a pull action instead of metadata_read.
When not authenticated GET methods return

Bearer realm="https://random-registry.com/oauth2/token",service="random-registry.com",scope="repository:my-registry/my-image:pull"

Solution.

Pass the resource url and use HEAD method to resolve the expected scope/action.When 405 is received use GET (small improvement of the reverted fix)
Pass the resource url and use GET method to resolve the expected scope/action. (like the revert fix but using only GET)
Check if current registry is ACR then query the resource url and use HEAD
Check if current registry is ACR then hard-code the action "scope="repository:xyz123:pull,metadata_read"" (like it is currently done with pull only)
any other ? @rarkins @viceice

vlagorce · 2022-04-26T08:23:38Z

Hi @rarkins @viceice , any preferred solution ?

rarkins · 2022-04-26T10:28:28Z

Which cases of registry need this and are we confident it will solve them all?
Maybe this simpler approach is better?

vlagorce · 2022-04-26T10:59:04Z

Does't no seems to be supported by many registries. The 405 fallback will be used in most cases. On the other hand GET is always available. There is not a lot of sens to have this overhead.
I prefer this one too. It keeps the flexibility to support dedicate scope/action discovering for other resources.

rarkins · 2022-04-26T13:17:35Z

@viceice are you OK with this proposal?

viceice · 2022-04-26T13:35:48Z

I'm ok with 2. that can also return a 200, so we should suppress further credential handling in that case , so that the following normal get call will use the already cached get result.

renovate-release · 2022-05-19T09:30:08Z

🎉 This issue has been resolved in version 32.64.0 🎉

The release is available on:

GitHub release
32.64.0

Your semantic-release bot 📦🚀

vlagorce added priority-5-triage status:requirements Full requirements are not yet known, so implementation should not be started type:bug Bug fix of existing functionality labels Mar 17, 2022

vlagorce changed the title ~~ACR~~ ACR Datasource unauthorized while fetching tags list. Mar 17, 2022

viceice added datasource:docker priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others and removed priority-5-triage labels Mar 17, 2022

vlagorce mentioned this issue Mar 17, 2022

fix(acr): Support specific ACR scope to fetch docker image tags. #14711

Closed

6 tasks

vlagorce mentioned this issue Mar 21, 2022

fix(docker): use a HEAD request to the real resource auth. #14744

Merged

6 tasks

rarkins closed this as completed in #14744 Apr 6, 2022

This comment was marked as outdated.

Sign in to view

viceice mentioned this issue Apr 8, 2022

fix(docker): Revert "fix(docker): use a HEAD request to the real resource auth." #15018

Merged

rarkins reopened this Apr 8, 2022

viceice mentioned this issue Apr 8, 2022

404 while listing Docker images with Renovate 32 and Sonatype NXRM 3 #15016

Closed

vlagorce mentioned this issue Apr 27, 2022

fix(docker): use a GET request to the real resource auth. (#14744) #15312

Merged

6 tasks

rarkins closed this as completed in #15312 May 19, 2022

github-actions bot locked as resolved and limited conversation to collaborators Jun 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ACR Datasource unauthorized while fetching tags list. #14708

ACR Datasource unauthorized while fetching tags list. #14708

vlagorce commented Mar 17, 2022 •

edited

viceice commented Mar 17, 2022

vlagorce commented Mar 17, 2022

viceice commented Mar 18, 2022

rarkins commented Mar 18, 2022

vlagorce commented Mar 21, 2022

viceice commented Mar 21, 2022

vlagorce commented Mar 21, 2022

vlagorce commented Mar 21, 2022

This comment was marked as outdated.

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

vlagorce commented Apr 8, 2022 •

edited

viceice commented Apr 8, 2022

vlagorce commented Apr 8, 2022 •

edited

vlagorce commented Apr 11, 2022 •

edited

vlagorce commented Apr 26, 2022

rarkins commented Apr 26, 2022

vlagorce commented Apr 26, 2022

rarkins commented Apr 26, 2022

viceice commented Apr 26, 2022

renovate-release commented May 19, 2022

ACR Datasource unauthorized while fetching tags list. #14708

ACR Datasource unauthorized while fetching tags list. #14708

Comments

vlagorce commented Mar 17, 2022 • edited

How are you running Renovate?

If you're self-hosting Renovate, tell us what version of Renovate you run.

Please select which platform you are using if self-hosting.

If you're self-hosting Renovate, tell us what version of the platform you run.

Was this something which used to work for you, and then stopped?

Describe the bug

Relevant debug logs

Have you created a minimal reproduction repository?

viceice commented Mar 17, 2022

vlagorce commented Mar 17, 2022

viceice commented Mar 18, 2022

rarkins commented Mar 18, 2022

vlagorce commented Mar 21, 2022

viceice commented Mar 21, 2022

vlagorce commented Mar 21, 2022

vlagorce commented Mar 21, 2022

This comment was marked as outdated.

viceice commented Apr 8, 2022

viceice commented Apr 8, 2022

vlagorce commented Apr 8, 2022 • edited

viceice commented Apr 8, 2022

vlagorce commented Apr 8, 2022 • edited

vlagorce commented Apr 11, 2022 • edited

vlagorce commented Apr 26, 2022

rarkins commented Apr 26, 2022

vlagorce commented Apr 26, 2022

rarkins commented Apr 26, 2022

viceice commented Apr 26, 2022

renovate-release commented May 19, 2022

vlagorce commented Mar 17, 2022 •

edited

vlagorce commented Apr 8, 2022 •

edited

vlagorce commented Apr 8, 2022 •

edited

vlagorce commented Apr 11, 2022 •

edited