Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker run doesn't pull down latest image if the image exists locally #13331

Closed
mlehner616 opened this issue May 19, 2015 · 89 comments
Closed

docker run doesn't pull down latest image if the image exists locally #13331

mlehner616 opened this issue May 19, 2015 · 89 comments

Comments

@mlehner616
Copy link

@mlehner616 mlehner616 commented May 19, 2015

docker run should have an --pull or --update option to pull down the most up to date version of an image tag, then run. The behavior should match docker build.

See #4238

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented May 20, 2015

I think it would be consistent with docker build --pull, on the other hand; the --pull flag had to be implemented because for build there's no alternative way to deal with this, see #4238 (comment). For docker run, this would work: docker pull myimage:tag && docker run myimage:tag

@unclejack you implemented this for docker build at the time, any thoughts?

@mlehner616

This comment has been minimized.

Copy link
Author

@mlehner616 mlehner616 commented May 20, 2015

I am actually using this work-around and generally agree. However, it comes down to UX and consistency. The very same argument could be made for the --no-cache flag where you could just docker rmi myimage:tag && docker build -t myimage:tag . So why does --no-cache exist? While they appear to be effectively doing the same, it's not necessarily clear to the user if docker is performing other optimizations with the --no-cache flag set that wouldn't happen using the workaround. A simple flag to pull is a lot cleaner.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented May 20, 2015

I agree on the UX perspective. But we need to prevent adding features "just because we can". Not entirely against, so don't give up hope yet. Let's see what others think.

Wrt docker rmi myimage:tag && docker build -t myimage:tag .. I don't think that's comparable. Doing that won't work if the old myimage:tag is still in use by a container, while --no- cache does. But I understand why you're comparing.

@stevenschlansker

This comment has been minimized.

Copy link

@stevenschlansker stevenschlansker commented May 22, 2015

IMO this is a good indication that tags should be immutable -- especially once caching mirrors start to come into play, the behavior of changing tags becomes incredibly complex and hard to manage.

Additionally, one of the "big selling points" of Docker is that when I docker run myapp:v3.1.4, I get the exact same image as anybody else doing so. Mutable tags breaks this promise. Anecdotally, we already were bitten by this once when an upstream image of etcd overwrote a release tag with a "fixed" version that introduced a regression.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented May 22, 2015

@stevenschlansker I agree, but changing tags to be immutable will not be possible without causing a serious BC breaking change.

However, #10740 and #11109 added support for referring images by digest, which does make this possible.

@eburnette

This comment has been minimized.

Copy link

@eburnette eburnette commented May 28, 2015

I'd really like to see this feature too. After rebuilding, I'd like to send a single docker command to remote hosts to start a container, but currently I have to send two: docker pull and docker run.

I found it surprising that docker didn't check for new versions by default. Currently, if the image doesn't exist, it pulls the newest version but from then on it doesn't check or do a pull. Was it done that way for performance reasons?

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented May 28, 2015

I found it surprising that docker didn't check for new versions by default. Currently, if the image doesn't exist, it pulls the newest version but from then on it doesn't check or do a pull. Was it done that way for performance reasons?

I think this was discussed at some point, but (if I recall correctly) there were a number of reasons;

  • If I have a locally built image (foobar:latest), and want to test it before pushing, automatically pulling that image from the registry would overwrite the local image.
  • Performance; pulling for each run can have quite an performance impact.
@sielaq

This comment has been minimized.

Copy link

@sielaq sielaq commented May 29, 2015

+1
@mlehner616 has mentioned

docker run should have an --pull or --update option

which imply that user have to use this option by purpose (it would not be a default option)
and have to deal with any consequences
(so mentioned by @thaJeztah reasons are valid, but if optional and documented - not so important).

This option is very needed,
imagine that we have to deal with hundreds docker slaves
if you prefer "consistence of images" over "speed",
then performance can be solved with local registry.

Sync problem we have solved using mcollective or consul exec to trigger docker pull
whenever new image is released - but not everyone have/use those tools.

@eburnette

This comment has been minimized.

Copy link

@eburnette eburnette commented May 29, 2015

@thaJeztah Yes, I can understand now why it was done this way by default, but it was surprising at first. Docker run might get you the latest image or it might get you a year old image, you can't tell.

@rbair23

This comment has been minimized.

Copy link

@rbair23 rbair23 commented Jun 12, 2015

I was just bitten by this. From an API perspective, the mistake the Docker team made was adding the shortcut to "run" to auto-pull images. The problem is that pulling on "run" becomes a modal operation. How the run command behaves w.r.t. pulling images has 2 modes: it behaves one way if the image has not been previously pulled and another way if it has. This of course is not naturally visible in the API -- you have to read documentation (or spend time debugging as I did and hit the internet) to learn about this mode in the API.

If the run command never auto-pulled, then the API would no longer be modal and it would be predictable (yes the simple intro example is 2 lines now instead of 1 but everybody knows what is going on and how things work and that is a really big deal for usability).

Or you have to make it so that by default "run" always pulls. I can see both reasons that @thaJeztah gives as good reasons not to want to do this.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jun 12, 2015

ping @duglin @tiborvass @unclejack (just some random names, sorry) wdyt? I think this is a reasonable request that brings a friendlier UX and parity with docker build --pull.

@adamkdean

This comment has been minimized.

Copy link

@adamkdean adamkdean commented Jun 12, 2015

Just my 2¢, how exactly is docker pull image && docker run image supposed to work with Swarm? I really need a way to pull the new image as part of the run command.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

@cpuguy83 cpuguy83 commented Jun 12, 2015

@rbair23 Actually the API does not automatically pull, this is strictly a client-side implementation, and the right one IMO.

@adamkdean
To get around this (and in general is good practice anyway), people should take advantage of the new ability to specify an image via it's content addressable digest.
https://docs.docker.com/reference/commandline/cli/#listing-image-digests
Though the UI (both in Hub and the CLI) could be better here, it's a great feature.

@rbair23

This comment has been minimized.

Copy link

@rbair23 rbair23 commented Jun 12, 2015

@cpuguy83 I was using API in the general term applying it to the programming interface exposed by the 'docker' command, not the REST API.

@adamkdean

This comment has been minimized.

Copy link

@adamkdean adamkdean commented Jun 12, 2015

@cpuguy83 I got around this by going though each Swarm node and running docker -H $host pull <image> prior to the run, but if I can specify a sha1 digest then that might actually be better. Thanks.

@tiborvass

This comment has been minimized.

Copy link
Collaborator

@tiborvass tiborvass commented Jul 24, 2015

@thaJeztah I think this is not a feature, it's a bug. With the ongoing trust efforts I hope that this will be resolved.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Jul 24, 2015

@tiborvass more an "omission"; I think the default behavior does make sense (see #13331 (comment)), but adding a --pull option is something that needs to be added.

Now that you completed the "docker daemon" PR, are you perhaps interested in creating a PR? 😉

Wrt the reference to "trust"; would that mean that on every build, Docker should pull / verify the image?

@ledjon

This comment has been minimized.

Copy link

@ledjon ledjon commented Jul 30, 2015

I'm not really concerned if it is part of default behavior or a --pull parameter, but I really think there should be a way to do this as part of one command. Perhaps even just spit out a message saying "I am starting this container, but be aware that this is not the latest. use --pull to get the latest"

Just my thoughts on it.

@andrerom

This comment has been minimized.

Copy link

@andrerom andrerom commented Aug 6, 2015

+1 on this, and +1 on adding this to docker-compose where you just like build don't know about external deps.

@coderfi

This comment has been minimized.

Copy link

@coderfi coderfi commented Aug 8, 2015

--pull would be huge!

It is too cumbersome, for every Dockerfile, to:

  • figure out what the base image is (awk the FROM line? hard code it in your build script?)
  • issue a docker pull for that if it contains the word 'latest' or is omitted (and/or leveraging additional build/CI systems)
  • then proceed to build

A generic option like --pull, --pull-always, --pull-if-missing, etc. would be much simpler than trying to do the above in every project.

Yes, you could argue: "well why not just specify the image version".

The problem with that is the case when you are maintaining a series of Dockers which build upon each other.

We'd have to roll in a complex build system to propagate base versions down as they get build up ... and that would mean hacking the downstream Dockerfiles each time... (or dynamically generating them? even messier...)

It is much simpler to just use the 'latest' for every FROM base in our Dockerfiles.

@andrerom

This comment has been minimized.

Copy link

@andrerom andrerom commented Aug 9, 2015

@coderfi This is for docker run, support for pull already exists for docker build which is what you are most likely looking for given you refer to FROM lines in DockerFile.

The need to also add this to docker run is most clear for tools like Docker compose being able to very simple being able to add such features for docker-compose up which would be huge, and is much needed.

@bmannix

This comment has been minimized.

Copy link

@bmannix bmannix commented Sep 18, 2015

+1 for a --pull flag. This would simplify my workflows.

@yinchuan

This comment has been minimized.

Copy link

@yinchuan yinchuan commented Oct 13, 2015

+1 for a --pull flag

@coolljt0725

This comment has been minimized.

Copy link
Contributor

@coolljt0725 coolljt0725 commented Oct 14, 2015

I think this is a easy client only implementation.
docker run will call createContainer which will pull the image.

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented Oct 14, 2015

@coolljt0725 sounds good, are you interested in opening a PR?

@coolljt0725

This comment has been minimized.

Copy link
Contributor

@coolljt0725 coolljt0725 commented Oct 14, 2015

PR is coming :)

@bean5

This comment has been minimized.

Copy link

@bean5 bean5 commented Dec 24, 2017

@NicoTexas

This comment has been minimized.

Copy link

@NicoTexas NicoTexas commented Feb 1, 2018

Rather than an option, I think a convention would be better, like the Maven SNAPSHOT one.
I am thinking of compatibility with Compose for example, but also that I would turn on this option almost everywhere, which at the end would create more risks and break the immutability contract of Docker images.

@bean5

This comment has been minimized.

Copy link

@bean5 bean5 commented Feb 2, 2018

@purn3ndu

This comment has been minimized.

Copy link

@purn3ndu purn3ndu commented Aug 23, 2018

Is this feature included /developed in the current version?

Or do we still have to
docker stop container_name
docker rm -v container_name
docker run --name container_name …
?

@cpuguy83

This comment has been minimized.

Copy link
Contributor

@cpuguy83 cpuguy83 commented Aug 23, 2018

No.

@zmackie

This comment has been minimized.

Copy link

@zmackie zmackie commented Nov 6, 2018

I’ve got a PR open to the CLI that should address this, if anybody wants to review: docker/cli#1498

@rishiloyola

This comment has been minimized.

Copy link

@rishiloyola rishiloyola commented Jan 15, 2019

any updates on this issue?

@Dmitry1987

This comment has been minimized.

Copy link

@Dmitry1987 Dmitry1987 commented Feb 11, 2019

The default behavior of docker run right now in Docker Version 18.03.1-ce does not pull newer image from the repo :O is this for real?

@Dmitry1987

This comment has been minimized.

Copy link

@Dmitry1987 Dmitry1987 commented Feb 11, 2019

it should always check if the tagged image changed in the repository (what if an urgent security fix was patched to the image, and it's expected to be distributed to all infra at the same time?), does it even makes sense that by default Docker does not check the image in the repo? That is SO weird.

@koenlek

This comment has been minimized.

Copy link

@koenlek koenlek commented Feb 11, 2019

There are some good reasons not to have it pull by default (or at least to expose both the options , i.e. to auto-pull or not to the user):

  • you often don't want to pull latest tags, because it can include breaking changes that you need to check and prepare for, before applying them in your production environment. Note that latest tags are regular tags, there is nothing special about them compared to other tags (afaik). It's just a best practice, but there is no guarantee whatsoever on how a maintainer follow the 'latest' best practice (e.g. will it always point to latest stable image? or might it also point to latest overall image, including pre-releases?).
    • Some popular images in docker-compose setups, like e.g. webhippie/mariadb, manage to only have a latest tag and make braking changes to them. You should be very careful with accidentally pulling them. It happened to me once that I pulled a newer version of webhippie/mariadb:latest, which contained a major mariadb upgrade (according to semantic versioning) with braking changes that broke my setup. You'd better have db backups before that happens to you ;)
  • Some maintainers manage to make breaking changes even in the same tags (even if it's not a latest tag). I experienced that with the official owncloud images before.

Bottom line: you first want to make sure that you trust a specific maintainer to provide you with a safely auto-pullable tag and/or have proper backup practices in place before you enable auto-pull on docker run in many cases.

Lastly: even if you'd want to change the default, please consider that you might get a whole lot of angry admins after you after they realize that suddenly the pulling behavior changed and started breaking there setups. Some admins consciously rely on the assumption that new version of a tag won't be pulled automatically.

@kwerle

This comment has been minimized.

Copy link

@kwerle kwerle commented Feb 11, 2019

I don't want it pull by default. I want the option to make it always check when it's run.

@Dmitry1987

This comment has been minimized.

Copy link

@Dmitry1987 Dmitry1987 commented Feb 12, 2019

@koenlek I agree with the last statement that there are many who use a random tag that 'just works' and will be surprised by the sudden change, in docker, especially if they also have apt/yum auto update turned on and one morning their docker version on all hosts gets auto-upgraded AND it pulls all images because of the auto-pull feature 🤣 ahahaha

@Dmitry1987

This comment has been minimized.

Copy link

@Dmitry1987 Dmitry1987 commented Feb 12, 2019

But I still want the auto-pull flag in docker! Very surprised that it doesn't exist.. sounds more like a feature that had to be there at v0.0.2 release, and not discussed during v1.19.X release, that's crazy.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

@cpuguy83 cpuguy83 commented Feb 12, 2019

Ultimately this is a docker/cli issue.
I recall there was a pr for this awhile ago which got stalled on UX. Might be worth digging through that again.

@zmackie

This comment has been minimized.

Copy link

@zmackie zmackie commented Feb 12, 2019

cough #13331 (comment)

@cpuguy83

This comment has been minimized.

Copy link
Contributor

@cpuguy83 cpuguy83 commented Feb 12, 2019

Ah, thanks! Didn't realize this got submitted. Sorry for the slow turn-around.

@Dmitry1987

This comment has been minimized.

Copy link

@Dmitry1987 Dmitry1987 commented Feb 12, 2019

@Zanadar is there a way to vote on the PR so it gets attention from the maintainers? If there is, you can count on me 👍 +1 , just tell me where to click

@zmackie

This comment has been minimized.

Copy link

@zmackie zmackie commented Feb 12, 2019

@Dmitry1987 feel free to click through on that PR and upvote there

@zmackie

This comment has been minimized.

Copy link

@zmackie zmackie commented May 17, 2019

@mlehner616 #13331 (comment) has been merged and should close this issue (I believe).

@thaJeztah

This comment has been minimized.

Copy link
Member

@thaJeztah thaJeztah commented May 17, 2019

Ah, yes, docker/cli#1498 has been merged, but missed the cut-off for Docker 19.03, so will ship as part of Docker 19.09 (but you can download nightly builds with that change)

Let me close this one

@fescobar

This comment has been minimized.

Copy link

@fescobar fescobar commented Nov 22, 2019

#!/bin/bash

# This is not the best option to pull the latest images but it's a workaround.
# When you try to pull an image with a tag that already exists in your machine, Docker by default doesn't pull the image.
# With this workaround, I'm comparing the Digest (sha256) what I have locally against what I have in my docker hub remotely.
# If there is a difference, then I remove my local image and I will pull the latest from the remote hub.
# Otherwise I will keep using the same image existing in my machine. 

# EXAMPLE:
DOCKER_REGISTRY=docker.io
DOCKER_NAMESPACE=frankescobar
DOCKER_IMAGE=allure-docker-service
DOCKER_TAG=2.13.0

# Use 'DOCKER_ENCODED_AUTH' for private repositories. Use user:pass encoded in base64. 
# Example
# frank:somepass
# ZnJhbms6c29tZXBhc3M=
DOCKER_ENCODED_AUTH=ZnJhbms6c29tZXBhc3M=


DOCKER_API_TOKEN=$(curl -X GET "https://auth.docker.io/token?service=registry.$DOCKER_REGISTRY&scope=repository:$DOCKER_NAMESPACE/$DOCKER_IMAGE:pull" --header "Authorization: Basic $DOCKER_ENCODED_AUTH" | python -c "import sys,json; print json.load(sys.stdin)['access_token']")
REMOTE_TAG_SHA_DIGEST=$(curl -X GET https://registry.hub.docker.com/v2/$DOCKER_NAMESPACE/$DOCKER_IMAGE/manifests/$DOCKER_TAG --header "Authorization: Bearer $DOCKER_API_TOKEN" --header "Accept: application/vnd.docker.distribution.manifest.v2+json" | python -c "import sys,json; print json.load(sys.stdin)['config']['digest']")


LOCAL_TAG_SHA_DIGEST=$(docker inspect --format='{{index .Id}}' $DOCKER_NAMESPACE/$DOCKER_IMAGE:$DOCKER_TAG) || true


if [ -n "$LOCAL_TAG_SHA_DIGEST" ] ; then
    if [ "$LOCAL_TAG_SHA_DIGEST" != "$REMOTE_TAG_SHA_DIGEST" ]; then
        docker image rm -f $DOCKER_NAMESPACE/$DOCKER_IMAGE:$DOCKER_TAG
    fi
fi

docker pull $DOCKER_REGISTRY/$DOCKER_NAMESPACE/$DOCKER_IMAGE:$DOCKER_TAG

Requirements:

  • Python
    Note:
  • It will pull the image even if appears an error.
nestorzaburannyi added a commit to nestorzaburannyi/annotate that referenced this issue Jan 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.