Update with 'Pull latest image version' doesn't work with an authentified registry

[githubbab]

Bug description
When container's update with 'Pull latest image version' enable, image was not pull. (rejected)
Registry log : error authorizing context: authorization token required
If i delete service and update the stack, that works.

Expected behavior
Portainer must pull new image

Steps to reproduce the issue:

1. have an authentified registry configured in portainer
2. Go to a stack
3. select a service with image in the registry
4. click on update
5. select 'Pull latest image version'
6. validate
7. the container not run => rejected

Technical details:

• Portainer version: 2.13.1 CE
• Docker version (managed by Portainer): 20.10.14, build a224086
• Platform (windows/linux): Debian Buster 10
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): docker-compose run -d
• Browser: Chrome
• Use Case: Using Portainer at Education
• Have you reviewed our technical documentation and knowledge base? Yes

[nickwest]

I've encountered the same issue and verified that this issue does not exist on the swarm itself when using the --with-registry-auth flag. It seems like the --with-registry-auth flag is potentially not being applied to portainer's docker commands where it is required (those operating on a private registry)

[ant32t]

After updating to 2.13.1 we have the same issue I believe. Using a webhook we get this respone

{"message":"Error pulling image with the specified tag","details":"Error response from daemon: Head https://registry.gitlab.com/v2/***/***/***/manifests/***: denied: access forbidden"}

going to image list in portainer and pulling the same image works fine.

[lloydpick]

Having same/similar issue with a private registry, even though I have it configured in Portainer, everytime I restart a service it rejects all deployments with No such image.

I have to then get Portainer to pull the image manually (through the images tab), and then the service will start. However this is causing issues as it means things can't balance as unless I manually pull the image to each host it's ineligable to start there.

I'm using version 2.13.1.

[ant32t]

We downgraded to 2.11.1 and everything has been working great since.

[samdulam]

We have a fix for this in the works to be included in our 2.14 release.
You could try it now by using portainerci/portainer:pr7037 image.

Please keep in mind it is a development build and should not used in Production.

[jtraxy]

As @samdulam mentions, there is a fix around this is in the new 2.14 release, which is now available.
It would be good if you could confirm that it resolved your exact issue though, as the details are a bit different.
#7095

[huib-portainer]

Yes, this has been resolved in 2.14: #7095

[yannikhaffke]

Hello,
I currently face the same problem. However, I am currently on the release 2.14.1. Should not the problem have been solved with this version?

I am running a 3 node Docker Swarm. The portainer volume is placed on a nfs share. Pulling the images manually via the "Images" button works fine with the configured registry (I’m using authenticated DockerHub). But now I need to download all images to all 3 nodes manually, which is really annoying. After that I can deploy / update the stack without any problem.
I noticed there is no option to select the registriy to use when deploying or updating the stack. Maybe I am missing something but I couldn’t find any hints in the documentation.

Technical details:

Portainer version: 2.14.1 BE
Docker version: 20.10.14
Platform (windows/linux): VMware Photon OS 4.0
Browser: Firefox
Use Case: Using Portainer at Education
Have you reviewed our technical documentation and knowledge base? Yes

Also my system is showing the following strange behavior:
Endless refresh circle is showing:

Endless “Image up to date” is running and stuck:

Many thanks for your help.

[samdulam]

Share logs from each of the portainer agent containers, also look at updating to 2.15.1 if possible.

[yannikhaffke]

Hello @samdulam,

as soon as I try to start the stack I get the following error:

In the portainer_agent logs I can see the following error (I took a look via “docker service logs portainer_agent” -f):

portainer_agent.0.l3coc8aocmvb@docker-dp-e02    | 2022/09/21 09:09:52 http error: Unable to retrieve DockerHub token from DockerHub (err=Get "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull": dial tcp 44.205.64.79:443: i/o timeout (Client.Timeout exceeded while awaiting headers)) (code=500)
portainer_agent.0.l3coc8aocmvb@docker-dp-e02    | 2022/09/21 09:09:55 http error: Unable to retrieve DockerHub token from DockerHub (err=Get "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) (code=500)
portainer_agent.0.l3coc8aocmvb@docker-dp-e02    | 2022/09/21 09:10:24 http error: Unable to retrieve DockerHub token from DockerHub (err=Get "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull": context deadline exceeded (Client.Timeout exceeded while awaiting headers)) (code=500)

The error is created by the correct node on which the container is supposed to be started (the timestamp is different from the screenshot but the events occurred at the same time). The configured credentials for docker.io are correct. I can download the image by using the “Pull Image” feature under the “Images” button. I hope this information helps you.

[yannikhaffke]

Hello again @samdulam,

a few more things from my side:

1. I am using a proxy to access docker.io
2. I configured the proxy in /etc/systemd/system/docker.service.d/http-proxy.conf which works fine
3. Using "docker login" with the same credentials works fine
4. Using "docker pull" to download the same image as in portainer works fine
5. The images are quit large (around 700MB) might this be a problem?

I'm not able to find / think about any other configuration which could explain the problem.

Many thanks for your help.

[kasraJ]

Hello! I appreciate the hard work the Portainer team puts into maintaining the project. However, I’ve noticed that this issue still persists in version 2.19.4, and it can be quite frustrating. Could you kindly provide any information about the timeline for fixing this issue? Thank you

[jtraxy]

@samdulam, maybe reopen this issue or ask user to raise separate?
Comments here seem to suggest that EE-3308 in 2.14.0 didn't resolve the problem.
Cheers
James T.

[samdulam]

Hello! I appreciate the hard work the Portainer team puts into maintaining the project. However, I’ve noticed that this issue still persists in version 2.19.4, and it can be quite frustrating. Could you kindly provide any information about the timeline for fixing this issue? Thank you

@kasraJ
Is it possible to share the complete compose YAML that you see this issue with. You can sanitize any secrets/password etc.

Thanks