New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[1.12-rc] docker should have a way to pull/update service images #24066

Closed
thaJeztah opened this Issue Jun 28, 2016 · 50 comments

Comments

@thaJeztah
Member

thaJeztah commented Jun 28, 2016

It's currently not possible to force updating a service's image, for example, if a service uses nginx:1.11.1-alpine, and an updated version is available for that tag, it's not possible to force pulling the latest version.

Related; when deploying a service, we want to be sure that each node uses the exact same version of the image.

We briefly discussed this yesterday, and are thinking of having Docker determine the sha256/digest of the specified image, and use that to deploy the service. When updating a service, the sha256/digest of the image is checked again, to know if an updated version is available and should be pulled.

/cc @mgoelzer @aluzzardi @tonistiigi @tiborvass

@tonistiigi

This comment has been minimized.

Member

tonistiigi commented Jun 28, 2016

We briefly discussed this yesterday, and are thinking of having Docker determine the sha256/digest of the specified image, and use that to deploy the service. When updating a service, the sha256/digest of the image is checked again, to know if an updated version is available and should be pulled.

If registry supports schema2 we should add the image ID to the service spec as well. This will remove unneeded pulls when image was pushed again or moved between registries.

For better error reporting, maybe we should also force it to offline images as well. So if we can't pull the manifest on the current manager node, there is a check to see if the image exists on this same node. If it exists we fix it to a specific image ID, if not then service creation will fail.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Jul 5, 2016

If registry supports schema2 we should add the image ID to the service spec as well. This will remove unneeded pulls when image was pushed again or moved between registries.

We already have digests to support this. We should actually use them.

The bug here is that auto-pull wasn't added into the exec/container implementation integrated with the daemon. The bug here is that we have no way to force a re-launch of all tasks.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Jul 6, 2016

@thaJeztah Could you clarify?

Looking at the code, we should already be auto-pulling on task execution. Is the problem that you, as a user, have no way to forcing a re-pull to ensure the latest image?

You can force this by scaling down, then up. That will force all the tasks to be redistributed, pulling the new image.

@tiborvass tiborvass added priority/P2 and removed priority/P1 labels Jul 6, 2016

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 6, 2016

Is the problem that you, as a user, have no way to forcing a re-pull to ensure the latest image?

That's the primary issue; scaling down, then up feels like a really hacky way to ensure I get the latest image, also (I think) only the instances that were replaced will actually be updated, unless I scale down to zero.

The other discussion (but perhaps I should open that as a separate one), that, without using digest, there's no guarantee that all my replicas run the same version of an image. For example, docker service create --name foobar nginx:123 then, after a while docker service scale foobar=5, then the newly created replicas may be running on a different version than the old ones.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Jul 6, 2016

The other discussion (but perhaps I should open that as a separate one), that, without using digest, there's no guarantee that all my replicas run the same version of an image. For example, docker service create --name foobar nginx:123 then, after a while docker service scale foobar=5, then the newly created replicas may be running on a different version than the old ones.

That is the whole point of digests. They provide this guarantee, unless you don't change your tags. If you change the value of your tags, you don't get any guarantee that instances across nodes are running the same thing.

@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jul 7, 2016

@thaJeztah @stevvooe i'm confused. What about service update --image foo ? That will force a reevaluation of foo, resolving it on manager to a digest, and then all workers have the newly resolved digest. Best of both worlds everyone is happy.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 7, 2016

@tiborvass looks like service update --image foo doesn't trigger a pull;

/# docker service create --name bla nginx
59mh7b6i8mamagr1kb1r93led

docker service ls
ID            NAME  REPLICAS  IMAGE  COMMAND
59mh7b6i8mam  bla   1/1       nginx
/ # docker service update --image nginx bla
bla
/ # docker service ls
ID            NAME  REPLICAS  IMAGE  COMMAND
59mh7b6i8mam  bla   1/1       nginx

I think the issue is that no changes are detected, so it doesn't re-deploy the service. These are the daemon logs when running the docker service update;

DEBU[0071] Calling GET /v1.24/services/bla
DEBU[0071] Calling POST /v1.24/services/59mh7b6i8mamagr1kb1r93led/update?version=12
DEBU[0071] form data: {"EndpointSpec":{"Mode":"vip"},"Mode":{"Replicated":{"Replicas":1}},"Name":"bla","TaskTemplate":{"ContainerSpec":{"Image":"nginx"},"Placement":{},"Resources":{"Limits":{},"Reservations":{}},"RestartPolicy":{"Condition":"any","MaxAttempts":0}},"UpdateConfig":{}}
DEBU[0080] 11ce7aed4932: Initiating bulk sync with nodes []
@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jul 7, 2016

@aaronlehmann @stevvooe wouldn't you consider that as a bug?

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Jul 8, 2016

@tiborvass Nothing actually changes in the configuration, so no updates are made.

There are two three things we should look at here:

  1. Let's use a digest from the client side to set the image name. That will ensure the updates are really updates. We've discussed this and I think this is the right way to go.
  2. We should also have a way to force a redeploy. This could be done with a counter or random field. If you set it another value, it will force a redeploy of the whole task set without any actual changes. I think this will have general applicability for debugging and other undetectable changes. cc @aaronlehmann
  3. The output should make it clear whether or not there will be updates based on the changes.

Let me know what you think and I can start filing bugs.

@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jul 8, 2016

We are conflating two issues here. I consider this issue to be resolved by:

  • keeping current behavior of always pulling on service create
  • changing behavior of service update, so that it always pulls when --image is specified
  • making sure that the tag is resolved to a digest as early as possible, which for a 1.12 timeframe it means resolving on the manager the user is talking to. Client-side resolving is strictly equivalent to DOCKER_CONTENT_TRUST being set by default, which is orthogonal to this discussion.

What you're mentioning is an additional issue, which is having a way of forcing the rescheduling, when the environment changed or a node was added. I'm not sure we can have that for 1.12 but it is orthogonal to this issue that thaJeztah opened.

I put this in P2 because it can be considered a bug. The current behavior will never be the expected one.

@stevvooe feel free to open an issue for forcing rescheduling

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 8, 2016

I think this will have general applicability for debugging and other undetectable changes

Could be useful to force a re-balance (e.g. after adding a new node)

@mackeyja92

This comment has been minimized.

mackeyja92 commented Sep 1, 2016

Hope you don't mind if I chime in here. I have had to put the hash of the image at the end when doing service update --image. This make it use the new image but I have had to go to each swarm node and manually pull that image first.

# On all my nodes
docker pull mysite.com:4567/image

#Then on one node, get the hash
docker images mysite.com:4567/image
docker service update --image mysite.com:4567/image:7791fb4 my_service

Really annoying. It would be much better to have figure out the hash of the latest one automatically and have it pinned to that hash until next time.

docker service update --image mysite.com:4567/image:latest my_service

docker service ps my_service
>>> ID                         NAME               IMAGE                       NODE     DESIRED STATE  CURRENT STATE               ERROR
dzu508odhb63ird37bxxoyojc  my_service.1      mysite.com:4567/image:7791fb4    host        Running   Running 1 minutes ago

However, what I do not want is for it to update when I do any other type of update/scaling to the service. I want to have to tell it explicitly to update the image no matter what. Even when scaling up/down the new nodes should take after whatever image was specified when I ran the update or the initial service creation.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Sep 1, 2016

@mackeyja92 While the rest of your comment makes sense, I am confused why you have to go to each node to pull the image. On startup, tasks will always try to pull the latest for a given tag. Does that registry require a login? What error do you get when you don't pull all the images?

@tonistiigi

This comment has been minimized.

Member

tonistiigi commented Sep 1, 2016

@stevvooe @mackeyja92 I think mysite.com:4567/image:7791fb4 here is the repo:id syntax that we really should remove (see #22112). That reference is local only and not pullable. Instead run docker inspect -f '{{index .RepoDigests 0}}' localimage that gives you a digest reference that is securely pullable from the registry. Note that you only have these references for images pulled-by-digest or pulled with docker v1.12+.

@mackeyja92

This comment has been minimized.

mackeyja92 commented Sep 1, 2016

@stevvooe The first time I start the service it pulls the image fine, however when doing an update I have to go pull the latest images and then update.

@tonistiigi I will look into that.

update:

I have looked into it and it still doesn't automatically pull. I have to go to each node and pull before I can run update. Really preventing me from doing automatic deployments.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Sep 2, 2016

@mackeyja92 If you have the image set to mysite.com:4567/image:7791fb4, it is never going to update, since it is fixed to that image id. If you want it to update when a new task is created (during service creating or scaling), you'll have to specify the image as mysite.com:4567/image or simply get the new image id and use that.

If you're using the method suggested by @tonistiigi, you'll to run that command each time you want the repo to update. The workflow would be something like this:

docker pull mysite.com:4567/image # get the latest version of mysite.com:4567/image
docker service update --image $(docker inspect -f '{{index .RepoDigests 0}}' mysite.com:4567/image)

That will force all instances to be on the same version using a rolling update, verified by digest.

@mackeyja92

This comment has been minimized.

mackeyja92 commented Sep 2, 2016

@stevvooe Sorry, I think you may have misunderstood why I was using the ID. I understand that if I manually set it to a ID it will never update. I was only doing that because after going to every node and pulling the image and running the following it still would be on the old image.

docker service update --image mysite.com:4567/myimage my_service

What I was doing was essentially exactly the same as you were doing except that I was using the image ID instead of the digest hash. The problem that I wanted fixed is having to go to every machine and pull it before running the update command. I am perfectly happy using the RepoDigest instead of the ID if the machines will pull it automatically if they don't have it after running the update command.

@stevvooe

This comment has been minimized.

Contributor

stevvooe commented Sep 2, 2016

The problem that I wanted fixed is having to go to every machine and pull it before running the update command.

Ok, I'm very confused by this behavior. Do you have log messages from the node during the pull?

@t-my

This comment has been minimized.

t-my commented Sep 28, 2016

Suggesting that we change the default behaviour to that if the latest tag image have change, we do update. This way it's so much simpler to update with latest, rather than building CI to specify versions/tags.

@manast

This comment has been minimized.

manast commented Sep 28, 2016

@tsoikkel One of the problems in my opinion is that :latest is a tag that is standardized for representing the last version of an image, and I think most people (me included until very recently), thought that the semantics of latest are different than the one for other tags. This should be much better explained on the docs, but even an explanation will not suffice since it creates a natural expectation and behaviour.

@jaredk-beta

This comment has been minimized.

jaredk-beta commented Sep 29, 2016

I've been able to confirm that adding or updating a container label will trigger a redeployment and pull down the latest image. So this example will deploy an updated image without needing to change the tag, just by using an arbitrary label:

docker service update --container-label-add last_deployed="<timestamp>" service_name

That being said, I'd like to add my vote for making the default behavior to pull down the latest version when running docker service update --image username/image service. I expected this to be the case and was disappointed to find that it did nothing unless you changed the tag.

@manast

This comment has been minimized.

manast commented Sep 29, 2016

@kingjared hmm, wouldn't that behaviour be a bug? Lets say that I have actually not the :latest image but :1.0.0, I certainly wont like to have all my nodes to download again that same image just because I changed a label.

@jaredk-beta

This comment has been minimized.

jaredk-beta commented Sep 29, 2016

@manast My example above just causes Swarm to redeploy the containers for a service (since I made an arbitrary change to the container spec). If the image you're using hasn't updated it won't re-download anything.

@manast

This comment has been minimized.

manast commented Sep 29, 2016

@kingjared aha, thats quite interesting. I will integrate this tomorrow to see how it goes.

@t-my

This comment has been minimized.

t-my commented Sep 30, 2016

This should be default for all tags. If the image have changed for 1.0, it should be pulled and deployed.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Sep 30, 2016

@stevvooe @tonistiigi @tiborvass where are we on this one; I was thinking along these lines;

  • digests are the best (only?) option when deploying a service, because you want every replica/task to be on the same image version (to prevent surprises where different replicas/tasks show different behavior)
  • digests are not very friendly to read (foo@sha256:deadbeef has no meaning, whereas foo:v2.0.x does)
  • using digests, and updating services should be easy

My suggestion would be to;

  • Store both what's specified by the user ("friendly" / "human-readable" tag) and the digest in the Service specs. The "friendly" tag can be used for presentation (foo:v2.0.x (foo@sha256:deadbeef)), and to resolve the current digest for that tag, as tags are mutable
  • If the user/client specifies a digest for the image instead of a tag (e.g. if content-trust is enabled), both "friendly" and "digest" contain the same value. The image is never updated automatically, unless the service is updated to a different image / tag /digest.
  • the digest is resolved by the manager when creating or updating a service, and stored in the Spec; if the digest is updated, tasks need to redeployed (which should happen automatically, since the Spec is updated)

Does this sound right?

@Vanuan

This comment has been minimized.

Vanuan commented Sep 30, 2016

Seeing latest@timestamp while using digest internally would be great. As far as I understand, tags can be changed. So storing timestamp when the tag was last fetched with an ability to force fetching new version is essential.

The current workaround is to use version tags. But it doesn't work very well for continuous deployment.

Is that right?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Sep 30, 2016

Although I see where you're coming from, I don't think we should introduce additional concepts of @timestamp. The service definition already shows the created/updated timestamps (see docker service inspect);

        "CreatedAt": "2016-09-30T12:09:13.91371889Z",
        "UpdatedAt": "2016-09-30T12:09:13.91371889Z",

But it doesn't work very well for continuous deployment.

For continuous deployment, and if this is important to you , I'd recommend generating a tag that works for your use case.

@Vanuan

This comment has been minimized.

Vanuan commented Sep 30, 2016

I'm coming from Docker Hub automated builds.
My workflow is the following:

  • merge something to a "stable" branch
  • wait for hub to build a new "latest"
  • trigger docker service update

Are there any other recommended workflows?

@aaronlehmann

This comment has been minimized.

Contributor

aaronlehmann commented Oct 2, 2016

cc @nishanttotla, who is working on having Docker resolve image digests when creating or updating a service

@t-my

This comment has been minimized.

t-my commented Oct 3, 2016

My workaround atm:

Add random metadata during deployment like container tags:

docker service update --container-label-add deploy=$(date -u +%Y-%m-%dT%H:%M:%S) --image repo:5000/image service-name'
@aluzzardi

This comment has been minimized.

Member

aluzzardi commented Oct 3, 2016

@Vanuan @tsoikkel In order to have a guaranteed, reproducible deployment you should use digests.

Tags can change, however, digests are immutable - they're basically the checksum of a particular image build and will force docker to pull that exact version.

You can see digests during push and pull, example:

$ docker pull redis
Using default tag: latest
latest: Pulling from library/redis
Digest: sha256:38e873a0db859d0aa8ab6bae7bcb03c1bb65d2ad120346a09613084b49185912
Status: Image is up to date for redis:latest

You can also get it by inspecting:

$ docker inspect -f '{{ .RepoDigests }}' redis:latest
[redis@sha256:38e873a0db859d0aa8ab6bae7bcb03c1bb65d2ad120346a09613084b49185912]

Then you use it to deploy:

$ docker service create --name cache 'redis@sha256:38e873a0db859d0aa8ab6bae7bcb03c1bb65d2ad120346a09613084b49185912'

This guarantees that the same exact version gets pulled all the time.

If you make changes to the image then push it, just get the new digest and do a:

$ docker service update cache --image redis@sha256:NEW_DIGEST

Note for the future: We (specifically, @nishanttotla) are working on performing the digest resolution automatically for you from the CLI.

Which means that in 1.13 (hopefully), whenever you do a docker service update --image my-image, the CLI will automatically translate that to my-image@sha256:current_digest which will force a rolling update if the digest is different from the one currently deployed (but will result in a no-op otherwise).

One of the big advantages of doing it this way is that even if you push to the registry, all of your service instances will continue running the same image version even if Docker re-deploys a container to a different machine and has to pull again or if you decide to scale up etc. The only way to update a service's image after a push is to explicitly request it by doing a service update.

@justincormack

This comment has been minimized.

Contributor

justincormack commented Oct 3, 2016

I recommend not just using "latest" as you can't roll back to a previous
image or pull it for testing. You really need a unique tag for every image
you put into production.

I don't think there are well documented best practises here, will try to
write something up.

On 30 Sep 2016 3:04 p.m., "John Yani" notifications@github.com wrote:

I'm coming from Docker Hub automated builds.
My workflow is the following:

  • merge something to a "stable" branch
  • wait for hub to build a new "latest"
  • trigger docker service update

Are there any other recommended workflows?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#24066 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAdcPKqSiH3IeILM6TzNmXV8l5TQL66uks5qvRbpgaJpZM4JAiYw
.

@manast

This comment has been minimized.

manast commented Oct 3, 2016

@justincormack I have a workflow where one of the differences between production and development is the use of the latest tag. It is very handy to deploy to the dev environment the latest code without needing to create tags for it...

@marcellodesales

This comment has been minimized.

marcellodesales commented Nov 2, 2016

I'm looking for the similar workflow with the @Vanuan ...

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Nov 2, 2016

@marcellodesales @Vanuan there will be a --force option in docker 1.13; #27596 which will force updating the service (and pulling the latest version of the image)

@marcellodesales

This comment has been minimized.

marcellodesales commented Nov 2, 2016

@thaJeztah sounds great!! thanks for keeping us posted!

@nishanttotla

This comment has been minimized.

Contributor

nishanttotla commented Nov 11, 2016

After #28173 was merged, it is now possible to pin service images by digest. That is, creating a service with an image and tag will always resolve to a digest, and all tasks will use exactly the same image.

For a running service, doing an update with the --image tag such as docker service update --image name:latest will result in a re-resolution of the tag to digest, and a pull will occur if a new image was pushed under the same tag. On the other hand, if the --image tag is not specified, then an updated image will not be pulled, ensuring that all service tasks continue to run with exactly the same image. We'll work on improving the UX before the 1.13 release.

Please test out the master branch, or the upcoming RCs. Looking forward to feedback.
cc @marcellodesales @Vanuan @manast

@tonistiigi

This comment has been minimized.

Member

tonistiigi commented Nov 11, 2016

fixed by #28173

@tonistiigi tonistiigi closed this Nov 11, 2016

@bobbydeveaux

This comment has been minimized.

bobbydeveaux commented Dec 10, 2016

Hurrah. Just what I was looking for - thanks! :)

@xesina

This comment has been minimized.

xesina commented Jan 5, 2017

Great, thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment