Non-standard image refs does not map to expected repositories in config.json #796

PssbleTrngle · 2021-01-25T10:34:27Z

Describe the bug
I am unable to pull private repositories with mapped config.json

Unable to update container "/grafana_importer": Error response from daemon: pull access denied for dockergelb/grafana-importer, repository does not exist or may require 'docker login

To Reproduce
I have mapped the config.json in /root/.docker, to /config in the watchtower container, but it is still failing to pull private repositories, even though the same repos are accessible on the host system.

This is the docker-compose file I use, (.env just contains the notification variables):

version: "3.3"

services:
  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
    env_file: .env

Environment

Platform: Linux/Debian
Docker version: 20.10.2
Watchtower version: latest

The text was updated successfully, but these errors were encountered:

PssbleTrngle · 2021-01-25T11:06:50Z

In the debug log it says it is unable to find credentials in the config for the username 'dockergelb', but that is the one I used to log in. Do I have to specify it in the config.json somewhere for watchtower to realize it has to use those credentials?

PssbleTrngle · 2021-01-25T11:33:05Z

It seems to parse the registry incorrent? When checking for the image dockergelb/test:latest, it should extract and log the registry here, but the log says

No credentials for dockergelb found" config_file=/config.json

Shouldn't the server be something like https://index.docker.io and not dockergelb, which is the username?

PssbleTrngle · 2021-01-25T11:41:07Z

It works when I manually copy my auth token for index.docker.io into a new entry for my username

...
"auths: {
    "dockergelb": {
        "auth": "*************************************"
    }
}

piksel · 2021-01-25T14:56:06Z

Can you do a pull using that config outside of watchtower? (I mean without the added dockergelb entry)
Just sudo docker pull dockergelb/grafana-importer should be equivalent I think...

PssbleTrngle · 2021-01-25T16:36:50Z

Yes, it's no problem outside.
It should also not be, as it's a completely normal docker config generated using docker login

{
        "auths": {
                "https://index.docker.io/v1/": {
                        "auth": "***********"
                }
        }
}

The part which parses the server/registry from the image tag is wrong I think (here)

It should not work for any of the tags in the format:

username/repo
username/repo:tag
repo

Because it just splits at / and uses the first part, expecting the image tag to be in the format of registry/username/repo:tag

piksel · 2021-01-25T20:24:03Z

The ref should already be normalized at that point, which means that it should start with the hostname etc.
I don't have any private images on dockerhub, but I can set one one up to verify.

PssbleTrngle · 2021-01-25T21:41:35Z

It should not matter whether it's private or not, the debug log should print your username as the extracted server name either way when trying to look for credentials for the image tag

piksel · 2021-01-25T22:39:07Z

Yeah, I can confirm that it doesn't work with private dockerhub repos. Not sure why it's not using the normalized image ref.
However it works correctly if you use the full image name:

docker.io/dockergelb/test:latest

Maybe @simskij knows why the the common repo names aren't prepended when resolving credentials...?

simskij · 2021-04-22T13:19:45Z

Currently, as @piksel mentioned, you either need to use the REPO_* env vars or use the fully qualified image name. This is not great though, so this is probably something we should have a look at some point. 👍🏼 Thanks for reminding us, @PssbleTrngle!

lsaadeh · 2021-07-15T07:42:13Z

Hey @simskij, just to clarity as I'm facing the same issue. This should work:

have you config file in the following format (generated using docker login):
{ "auths": { "https://index.docker.io/v1/": { "auth": "***********" } } }
add the volume ${HOME}/.docker/config.json:/config.json
add the fully qualified image name for your repo: docker.io/lsaadeh1/test:latest

piksel · 2021-07-15T10:34:09Z

Hey @lsaadeh, just to clarify, do you still have the issue, or are you saying that the above is what worked for you? 😁

It looks correct afaict.

lsaadeh · 2021-07-15T11:00:06Z

Yes.. I'm still having the issue. =(

For whatever reason, I get a 401 error msg:

time="2021-07-15T21:38:36Z" level=warning msg="Could not do a head request for \"docker.io/lsaadeh1/swiss-knife:latest\", falling back to regular pull." container=/swissknife image="docker.io/lsaadeh1/swiss-knife:latest"

time="2021-07-15T21:38:36Z" level=warning msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:lsaadeh1/swiss-knife:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/swissknife image="docker.io/lsaadeh1/swiss-knife:latest"

time="2021-07-15T21:38:39Z" level=info msg="Unable to update container \"/swissknife\": Error response from daemon: pull access denied for lsaadeh1/swiss-knife, repository does not exist or may require 'docker login': denied: requested access to the resource is denied. Proceeding to next."

Though when I run docker login in its successful:

Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/<user>/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

piksel · 2021-07-15T12:06:21Z

Yeah, it does correctly use you credentials, but they are not authorized to pull from that repo for some reason:

scope=repository:lsaadeh1/:pull, error=insufficient_scope

Since you have modified the log to remove sensitive information, it's kind of hard to tell what is typos and what might actually be a problem, but it looks like there is a 1 in some image refs (lsaadeh1) and not in others...

Maybe there are several entries in /home/<user>/.docker/config.json?

lsaadeh · 2021-07-15T22:37:37Z

Sorry for messing that up ><! I've updated my last post with the original details.. they're not that sensitive anyway... There's only 1 entry in my config file, I can also pull the container on my machine manually

staser12 · 2021-08-08T15:15:50Z

I am having absolutely the same issue :(

time="2021-08-08T15:04:20Z" level=warning msg="Could not do a head request for \"sha256:3102f04c02668da3b42aafa658b1280e40edfb333cb7919a9df7379813480e5e\", falling back to regular pull." container=/amazing_archimedes image="sha256:3102f04c02668da3b42aafa658b1280e40edfb333cb7919a9df7379813480e5e"
time="2021-08-08T15:04:20Z" level=warning msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://auth.docker.io/token\\\",service=\\\"registry.docker.io\\\",scope=\\\"repository:library/sha256:pull\\\",error=\\\"insufficient_scope\\\"\"" container=/amazing_archimedes image="sha256:3102f04c02668da3b42aafa658b1280e40edfb333cb7919a9df7379813480e5e"
time="2021-08-08T15:04:22Z" level=info msg="Unable to update container \"/amazing_archimedes\": Error response from daemon: pull access denied for sha256, repository does not exist or may require 'docker login': denied: requested access to the resource is denied. Proceeding to next."

piksel · 2021-08-08T17:22:16Z

@staser12 No, that is because of #1050

image="sha256:3102f04c02668da3b42aafa658b1280e40edfb333cb7919a9df7379813480e5e"

piksel · 2021-08-08T17:29:03Z

@lsaadeh Please open up a discussion if you want any further help with your setup. This issue should be limited to watchtower not resolving the image names to config.json entries the way that is expected.

staser12 · 2021-08-08T17:39:58Z

@staser12 No, that is because of #1050

image="sha256:3102f04c02668da3b42aafa658b1280e40edfb333cb7919a9df7379813480e5e"

Thank you for the reply! However I do have all my images named as well as containers. So it is really strange...

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS               NAMES
ed80618ff093        containrrr/watchtower            "/watchtower --inter…"   3 minutes ago       Up 3 minutes        8080/tcp            watchtowertest_watchtower_1
686e43f620df        staser12/watchtowertest:0.0.17   "/bin/sh -c 'python …"   3 minutes ago       Up 3 minutes                            watchtowertest_watchtowertest_1

piksel · 2021-08-08T17:44:38Z

@staser12 The container that had such an image was called amazing_archimedes (probably auto-generated). But please create a discussion if you want further assistance, as this is getting off topic.

staser12 · 2021-08-08T17:51:34Z

@staser12 The container that had such an image was called amazing_archimedes (probably auto-generated). But please create a discussion if you want further assistance, as this is getting off topic.

ok, thanks a lot for your help! I will try to figure out where does this container come from. I will create a new discussion in case I cannot figure this out.

dschmidt · 2021-11-12T12:46:14Z

I'm having (kind of) the same issue with pulling images from docker hub.

After fiddling around with the config.json and image names I've found exactly one configuration that actually works:
I have to use docker.io/user/image as image name and my config needs to look like this:

{
	"auths": {
		"https://docker.io": {
			"auth": "base64(user:password)"
		}
	}
}

This is very confusing especially because the watchtower documentation says I'm supposed to use index.docker.io/.../...as image name and logging in via docker login uses https://index.docker.io/v1/ as key in the auths object in config.
I could definitely not make this work in any combination when using index.docker.io in either place or in both.
I pretty much tested out all combinations afaict and the only version that actually works is what I said in the beginning (using docker.io everywhere).

This is with watchtower 1.3.0.

piksel · 2021-11-12T13:50:32Z

I made some experiments to check the behaviour of watchtower in regards to the config.json credentials. First I started three containers with the following images (the same image, but with different repository hosts):

❯ docker inspect wt-test1 wt-test2 wt-test3 | jq '.[].Config.Image'
"piksel/wt-test"
"index.docker.io/piksel/wt-test"
"docker.io/piksel/wt-test"

Then I ran docker login -u piksel to authenticate. And tried running watchtower, and here is the important parts:

DEBU[0007] Trying to load authentication credentials.    container=/wt-test1 image="piksel/wt-test:latest"
DEBU[0007] No credentials for piksel found               config_file=/config.json

DEBU[0004] Trying to load authentication credentials.    container=/wt-test2 image="index.docker.io/piksel/wt-test:latest"
DEBU[0004] Loaded auth credentials for user , on registry index.docker.io/piksel/wt-test:latest, from file /config.json

DEBU[0001] Trying to load authentication credentials.    container=/wt-test3 image="docker.io/piksel/wt-test:latest"
DEBU[0001] No credentials for docker.io found            config_file=/config.json

None of the images could be updated, but the "most promising" one found the credentials, but claimed that the user was an empty string. Looking in my config.json it's clear why:

{
        "auths": {
                "https://index.docker.io/v1/": {}
        },
		"credsStore": "desktop.exe",  
        "experimental": "enabled"
}

It did add the entry, but the actual credentials was stored in the credential helper (that watchtower has no access to).
If I then removed the credsStore field from the config and ran docker login -u piksel again, there were an actual entry inside the https://index.docker.io/v1/ key.

Upon running watchtower again with the new config.json:

DEBU[0004] Trying to load authentication credentials.    container=/wt-test2 image="index.docker.io/piksel/wt-test:latest"
DEBU[0004] Loaded auth credentials for user piksel, on registry index.docker.io/piksel/wt-test:latest, from file /config.json

It could now authenticate and pull the latest image.
If I then changed the key inside config.json to https://docker.io/v1/ it instead worked correctly for the image docker.io/piksel/wt-test:

DEBU[0001] Loaded auth credentials for user piksel, on registry docker.io/piksel/wt-test:latest, from file /config.json
DEBU[0001] Got image name: docker.io/piksel/wt-test:latest

So, in conclusion:
If you are hosting the images on docker hub, either start your containers with the prefix index.docker.io/. or edit the entry in your config.json to read https://docker.io/v1/ (and launch the images with the prefix docker.io/).
If the credentials are found, but the username is empty, check for a credential helper in the config and remove it to be able to store the credentials using docker login.

dschmidt · 2021-11-16T20:26:23Z

I'm having (kind of) the same issue with pulling images from docker hub.

After fiddling around with the config.json and image names I've found exactly one configuration that actually works: I have to use docker.io/user/image as image name and my config needs to look like this:
{
	"auths": {
		"https://docker.io": {
			"auth": "base64(user:password)"
		}
	}
}
This is very confusing especially because the watchtower documentation says I'm supposed to use index.docker.io/.../...as image name and logging in via docker login uses https://index.docker.io/v1/ as key in the auths object in config. I could definitely not make this work in any combination when using index.docker.io in either place or in both. I pretty much tested out all combinations afaict and the only version that actually works is what I said in the beginning (using docker.io everywhere).

This is with watchtower 1.3.0.

Actually what's in the documentation works for me, I've clarified it here: #1132 (comment)

simskij · 2022-01-22T17:45:13Z

Actually what's in the documentation works for me, I've clarified it here: #1132 (comment)

🎉

PssbleTrngle added Priority: Medium Status: Available Type: Bug Unintended behavior labels Jan 25, 2021

piksel mentioned this issue Aug 8, 2021

Docker containers with hash-only image cannot be updated #1050

Closed

piksel changed the title ~~Cannot pull private repositories with mapped config.json~~ Non-standard image refs does not map to expected repositories in config.json Aug 8, 2021

piksel mentioned this issue Nov 12, 2021

docs: add note about docker hub private images #1132

Merged

simskij closed this as completed Jan 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Non-standard image refs does not map to expected repositories in config.json #796

Non-standard image refs does not map to expected repositories in config.json #796

PssbleTrngle commented Jan 25, 2021 •

edited

PssbleTrngle commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

piksel commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021 •

edited

piksel commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

piksel commented Jan 25, 2021

simskij commented Apr 22, 2021

lsaadeh commented Jul 15, 2021 •

edited

piksel commented Jul 15, 2021

lsaadeh commented Jul 15, 2021 •

edited

piksel commented Jul 15, 2021

lsaadeh commented Jul 15, 2021

staser12 commented Aug 8, 2021 •

edited by piksel

piksel commented Aug 8, 2021

piksel commented Aug 8, 2021

staser12 commented Aug 8, 2021 •

edited by piksel

piksel commented Aug 8, 2021 •

edited

staser12 commented Aug 8, 2021

dschmidt commented Nov 12, 2021

piksel commented Nov 12, 2021

dschmidt commented Nov 16, 2021

simskij commented Jan 22, 2022

Non-standard image refs does not map to expected repositories in config.json #796

Non-standard image refs does not map to expected repositories in config.json #796

Comments

PssbleTrngle commented Jan 25, 2021 • edited

PssbleTrngle commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

piksel commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021 • edited

piksel commented Jan 25, 2021

PssbleTrngle commented Jan 25, 2021

piksel commented Jan 25, 2021

simskij commented Apr 22, 2021

lsaadeh commented Jul 15, 2021 • edited

piksel commented Jul 15, 2021

lsaadeh commented Jul 15, 2021 • edited

piksel commented Jul 15, 2021

lsaadeh commented Jul 15, 2021

staser12 commented Aug 8, 2021 • edited by piksel

piksel commented Aug 8, 2021

piksel commented Aug 8, 2021

staser12 commented Aug 8, 2021 • edited by piksel

piksel commented Aug 8, 2021 • edited

staser12 commented Aug 8, 2021

dschmidt commented Nov 12, 2021

piksel commented Nov 12, 2021

dschmidt commented Nov 16, 2021

simskij commented Jan 22, 2022

PssbleTrngle commented Jan 25, 2021 •

edited

PssbleTrngle commented Jan 25, 2021 •

edited

lsaadeh commented Jul 15, 2021 •

edited

lsaadeh commented Jul 15, 2021 •

edited

staser12 commented Aug 8, 2021 •

edited by piksel

staser12 commented Aug 8, 2021 •

edited by piksel

piksel commented Aug 8, 2021 •

edited