Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker service update --image "could not accessed on a registry to record its digest" #34153

Closed
sylvainmouquet opened this issue Jul 18, 2017 · 14 comments

Comments

Projects
None yet
7 participants
@sylvainmouquet
Copy link

commented Jul 18, 2017

Hello,

I use docker swarm with

docker -v
Docker version 17.06.0-ce, build 02c1d87

My swarm is composed only of one node :
docker node ls 2c9zrmt995p3kxn0qi4sg2xyu * ${nodeName} Ready Active Leader

When i run docker service update --image i have this error :

docker service update --image=${REGISTRY}:5000/${IMAGE}:latest ${serviceName}
image ${REGISTRY}:5000/${IMAGE}:latest could not be accessed on a registry to record
its digest. Each node will access ${REGISTRY}:5000/${IMAGE}:latest independently,
possibly leading to different nodes running different
versions of the image.
{serviceName}
Since --detach=false was not specified, tasks will be updated in the background.
In a future release, --detach=false will become the default. 

The file /home/${USER}/.docker/config.json exists and contains the auth for the container.
When i run docker images, i see docker has been able to download images automatically for others services.
I think the difference is for this service i have downloaded manually the image and after i have create the service.

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Jul 18, 2017

In the same style, docker service has downloaded an image of 5 days ago but there is a new image every day in the registry.
${REGISTRY}:5000/${IMAGENAME} <none> c4dad25bfd4a 5 days ago 1.2GB
If i run docker pull i download the last version

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

When updating services that need credentials to pull the image, you need to pass --with-registry-auth. Images pulled for a service take a different path than a regular docker pull, because the actual pull is performed on each node in the swarm where an instance is deployed. To pull an image, the swarm cluster needs to have the credentials stored (so that the credentials can be passed on to the node it performs the pull on).

Even though the "node" in this case is your local node, swarm takes the same approach (otherwise it would only be able to pull the image on the local node, but not on any of the other nodes).

Setting the --with-registry-auth option passes your locally stored credentials to the daemon, and stores them in the raft store. After that, the image digest is resolved (using those credentials), and the image is pulled on the node that the task is scheduled on (again, using the credentials that were stored in the service).

Can you confirm if passing --with-registry-auth makes the problem go away?

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Jul 18, 2017

Ok i have not set --with-registry-auth in the command line but why the image downloaded by the service is not the last version ?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

but why the image downloaded by the service is not the last version ?

If the service failed to connect to the registry, it didn't resolve the latest digest, so uses any image with the specified image:tag that was already present on the host. Based on your description, docker service update did not pull a new image at all.

Also be aware that when updating the image without changing image:tag, the service may not update the definition (i.e., re-resolve the image), unless --force is set (but I'd have to double check).

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Jul 18, 2017

Yes it's true i have already gone this scenarios but this time i have installed our application on a new virtual machine. We have a script which is in charge to deploy the application. It runs 4 docker stacks with the command line docker stack deploy --with-registry-auth.... I use this script on another environnement which deploy every night our application and for this environnement i have one manager and 2 workers. The manager does not host containers.
With this same script i have deployed the same application but this time the images was not the latest.
Docker service has downloaded images but not the latest version. I am sure that the image has
been downloaded by docker service because the version is defined to <none>.

I use docker stack deploy and the option --force is not present to this command. https://docs.docker.com/engine/reference/commandline/stack_deploy/
Do you have a solution to force the downloading of the latest version every time with docker stack deploy?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jul 18, 2017

For docker stack deploy, there's a --resolve-image=always option, but IIRC, that's the default, so it should resolve the latest image (given that it has the right credentials);

With this same script i have deployed the same application but this time the images was not the latest. Docker service has downloaded images but not the latest version.

What user does the script run as, and is that use logged in? If the script runs as a different user, it will look in that user's home directory for the credentials

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Jul 18, 2017

I use the same user.

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Jul 28, 2017

same bug here #34299

@DBLaci

This comment has been minimized.

Copy link

commented Aug 1, 2017

No #34299 is not the same. The problem I write there is not because of inaccessibility of private registry images.

@sylvainmouquet

This comment has been minimized.

Copy link
Author

commented Aug 14, 2017

I close this post i will open a new with more information if the problem occurs agin

@majstrooker

This comment has been minimized.

Copy link

commented Oct 23, 2017

I still get the same error, i.e.:

image dockersamples/visualiser:stable could not be accessed on a registry to record
its digest. Each node will access dockersamples/visualiser:stable independently,
possibly leading to different nodes running different
versions of the image.

Even when using --resolve-image=always and --with-registry-auth. I'm literally following the instructions from https://docs.docker.com/get-started/part5/#add-a-new-service-and-redeploy. Everything was working perfectly before. Does someone have any other idea on how to solve this?

@XLMBuildManager

This comment has been minimized.

Copy link

commented Mar 23, 2018

@majstrooker I'm having the same issue.

@trajano

This comment has been minimized.

Copy link

commented Sep 28, 2018

@majstrooker @XLMBuildManager does it still deploy correctly even with that message? What I found was that it may be a timing issue with Sonatype nexus. I wonder if I can speed it up.

@trajano

This comment has been minimized.

Copy link

commented Sep 29, 2018

Actually something is really borky with Sonatype Nexus and docker there, I am not sure which part itself is broken, but I didn't have any issues with registry:2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.