New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose up doesn't pull down latest image if the image exists locally #3574

Closed
kag0 opened this Issue Jun 9, 2016 · 65 comments

Comments

Projects
None yet
@kag0

kag0 commented Jun 9, 2016

It would be nice if there were an option to check for new versions of images when running docker-compose up.

We already have this functionality with docker build --pull as was discussed here moby/moby#4238 and there is an open issue to bring --pull to docker run here moby/moby#13331.

I propose adding --pull to up to always attempt to pull a newer version of the images in the compose file.

@shin-

This comment has been minimized.

Member

shin- commented Jun 9, 2016

Is there a reason why docker-compose pull && docker-compose up is impractical?

@kag0

This comment has been minimized.

kag0 commented Jun 9, 2016

I think most of the points for/against are the same as the ones discussed in the issue for adding --pull to docker run. They range from ux and consistency, to ease of scripting/workflow, to swarm integration (out of curiosity, what does docker-compose pull do with swarm?).
I don't think it's a major issue, but something to consider. The same type of users who desired the feature elsewhere would likely also enjoy it here.

@decoomanj

This comment has been minimized.

decoomanj commented Jun 30, 2016

I'm trying to execute "docker-compose build", but it doesn't refresh the image referenced in the Dockerfile, except when you use --pull.

You can also build containers during the start with up --build. But the newest images will not be pulled. Can we expect sthg like "docker-compose up --build --pull" (or similar)? Maybe it makes sense to place it in the YML since not all build must be refreshed (cfr. local images).

@cressie176

This comment has been minimized.

cressie176 commented Oct 11, 2016

Instead of (or in addition to) adding --pull to the cli, what about adding something on the service definition in the docker-compose file?, e.g.

version: '2'

services:

  postgres:
    image: postgres
    container_name: postgres
    pull: true
    ports:
     - '5432:5432'

This way if there's a service I don't care about being latest and one I do, docker-compose won't waste time downloading images I'm not interested in

@blockloop

This comment has been minimized.

blockloop commented Feb 13, 2017

I came here looking for this feature because we use it in our production cluster of Kubernetes. There the tag is "imagePullPolicy" and it can be set to "IfNotPresent", "Always", or "Never." Something similar for a compose environment would be nice.

@PengBAI

This comment has been minimized.

PengBAI commented Mar 3, 2017

In our case, we need to rebuild base image everyday to make sure the latest dependencies updated for applications. Docker compose up to pull the latest image with the same tag is a nice feature. Why not !

@marcelovcpereira

This comment has been minimized.

marcelovcpereira commented Mar 22, 2017

Any news about this?

@sugun999

This comment has been minimized.

sugun999 commented May 12, 2017

Hi,

Any news about this issue?

@jonathan-kosgei

This comment has been minimized.

jonathan-kosgei commented May 14, 2017

+1

1 similar comment
@ivytribe

This comment has been minimized.

ivytribe commented May 23, 2017

+1

@shin-

This comment has been minimized.

Member

shin- commented May 23, 2017

As I mentioned earlier, docker-compose pull && docker-compose up is functionally identical. There's no good reason to add another flag for it.

@shin- shin- closed this May 23, 2017

@blockloop

This comment has been minimized.

blockloop commented May 23, 2017

Imagine that git didn't have pull because git fetch && git merge origin/master is functionally identical.

@telly23

This comment has been minimized.

telly23 commented Jun 8, 2017

Adding a pull: true tag could be useful for example if some of the images you use in your compose file are in your cache. docker-compose pull pull all of the images in your compose file, and that pull will fail if these images are in your cache but not in the repository.

@fulllove890

This comment has been minimized.

fulllove890 commented Jun 12, 2017

+1

@JustinLivi

This comment has been minimized.

JustinLivi commented Jun 23, 2017

One scenario in which docker-compose pull && docker-compose up becomes impractical is when you're using multiple docker-compose files. You can easily end up with a command like docker-compose -f docker-compose.test.yml pull && docker-compose -f docker-compose.test.yml up.

@xellsys

This comment has been minimized.

xellsys commented Jul 12, 2017

We have a scenario in which we develop locally and we would only want to pull some of the images from remote. The one (or more) developed locally should remain untouched. In that case we are obligated to build/pull images manually before running docker-compose up.
A pull: true would be beneficial.

@lig

This comment has been minimized.

lig commented Jul 25, 2017

@shin- what about rethinking your decision on this? I believe comments and reactions to them look self-descriptive enough.

@shin-

This comment has been minimized.

Member

shin- commented Jul 25, 2017

No, I'm good. This is an open-source project, if you disagree with the conservative approach we take when adding features, you're welcome to fork it.

@alwye

This comment has been minimized.

alwye commented Oct 6, 2017

Agree on adding either a flag to the docker-compose up command or a parameter to the configuration. We use a base image with additional configuration that tends to change frequently during the development process. We want to create a fool-proof environment, where a developer can simply run docker-compose up without debugging what they don't need to debug.

I actually hit this thread after my colleague was checking my pull request and said it broke the build. And the reason being is that the base image was simply missing a couple of packages - but got executed in the final Dockerfile. It would be a good feature, but apparently not something you used, @shin- .

We'd be happy if you introduce this feature in the new version.

@kgalli

This comment has been minimized.

kgalli commented Oct 19, 2017

@shin- I think @blockloop showed with the git example how convenient / useful the pull option is. Honestly I hope for such things it is not necessary to fork any project.

I totally understand the conservative approach but it feels like it is not even considered as an option anymore. Maybe it can be part of an upcoming version?

@pwFoo

This comment has been minimized.

pwFoo commented Oct 30, 2017

A pull: IfNotPresent would be nice. So it could be possible to use fallbacks like

  1. use local image
  2. pull if isn't local
  3. build if not able to pull
@jrich523

This comment has been minimized.

jrich523 commented Nov 17, 2017

@shin- you keep asking for a reason why the && method wont cut it, and my reason is this. I use it for a "app" image for testing (Puppet PDK/Onceover). The compose file is part of the template repo so when a puppet dev (really ops folks) needs to make a new module they fork that repo. Jenkins runs the image for merge validation on that module repo (internally we have a jenkins plugin that handles the update for the jobs.) Now the folks who use this wont be docker experts, and having to tell them to do the && is an extra step that they can (and probably will) screw up. I dont see why it would be hard or what disadvantage it would create but this reasoning seems like a worthwhile reason to add it. It helps us developers send things out that require less directions and steps..

the short of it is.... to protect against laziness

@ostrolucky

This comment has been minimized.

ostrolucky commented Nov 29, 2017

Here is better reason: && is synchronous. But docker-compose has built in great support for parallel executor which optimizes this stuff. docker-compose up --pull --build could start building image and run it as soon as it's pulled, instead of waiting for all images to be pulled and only then start building

@port22

This comment has been minimized.

port22 commented Dec 4, 2017

@shin is assuming one's using for ourselves on a very rare basis with rare updating Dockerfiles.
But if you update a Docker image every day that developers will use, it will rapidly lead to development problems if just one developer forgets to pull the image which he has to do every day (especially those who are not really familiar with docker, and that's actually not their business to know and remember it). Disaster programmed.
Is it a big problem to just add this option to the docker-compose.yml ?
I mean, it will not change other things, it just adds functionality. ..

it's the killer reason why I can't use docker-compose and need to write wrapper scripts with legacy docker run commands , but that's ugly.

@themanifold

This comment has been minimized.

themanifold commented Feb 27, 2018

Trying to figure the circumstances under which this ticket was closed - would someone please elaborate? If it's any help, I am pro adding a --pull flag to docker-compose up.

@agnjunio

This comment has been minimized.

agnjunio commented May 23, 2018

So, here's a good use case:

I implemented a CI for my microservices project, which pulls new images to a registry when we develop new features in the backend service. The frontend team (which knows little about docker) needs to have a way to bring up the entire backend stack on their local machines, and they rely on the most updated images. If something breaks, only then they will remember to pull the images.

Now this is what happened: An entire development sprint went to fail because someone forgot to update the backend images, and developed an entire feature based on an old version. Blame on frontend team, but this could be avoided with this functionality (which I'll be doing using wrapper scripts).

@shin-

This comment has been minimized.

Member

shin- commented May 23, 2018

@agnjunio That sounds really unfortunate, sorry. However, if the person forgot to run docker-compose pull, I'm not sure how them forgetting to use a hypothetical --pull flag is any less likely?

@agnjunio

This comment has been minimized.

agnjunio commented May 23, 2018

@shin- Sorry, I forgot to mention an important thing: The solution in my case is to have the tag pull: always inside the yaml, maybe inside image: options

@solsson solsson referenced this issue May 25, 2018

Merged

Pull selectively #13

@solsson

This comment has been minimized.

solsson commented May 25, 2018

@ET-CS from #3574 (comment):

I'd expect --build to bring everything to latest.

IMHO --build should pull; no need for any other flag or configuration. if you don't want it to pull then specify image version.

AFAIK that would block the use case with a build using a FROM that is the result of a depends_on build.

version: "3.4"
services:
  some-base-image:
    image: our/base-image
    build:
      context: ./base
  # This Dockerfile has FROM our/base-image
  coolthing:
    depends_on:
    - some-base-image
    build:
      context: ./coolthing

I'm in favor of a flag as suggested in #3574 (comment) and #3574 (comment)

@ET-CS

This comment has been minimized.

ET-CS commented May 25, 2018

@solsson
I am not sure I see why or what it's blocking in this use case.
please share more information about that if you can.

@ruanltbg

This comment has been minimized.

ruanltbg commented May 29, 2018

@shin- The difference here would be that if I run docker-compose up --help I would receive a descriptive way in how to use the :latest image, instead nowadays I have to search in the doc or in the google that leads me to this thread and I need have to read all the comments to understand that docker-compose up does not do what I need/want, so now I need to run two command.

@agnjunio That sounds really unfortunate, sorry. However, if the person forgot to run docker-compose pull, I'm not sure how them forgetting to use a hypothetical --pull flag is any less likely?

Cordially

@ahannigan

This comment has been minimized.

ahannigan commented Jun 1, 2018

One of the main reasons we use docker-compose is the ability to place a docker-compose.yaml file in to a repository and have reproducible output when a developer pulls the repository and runs docker-compose up [service].

We use several services within our docker-compose files that perform tasks like running codegen and running a tool to dereference a JSON schema in to a single file.
Making sure these tools are up-to-date is critical especially if we update our codegen image to fix some common critical issue seen across all projects.

Having the ability to place:

services:
  codegen:
    image: myimage:latest
    pull: Always

would retain our ability to have a developer reliably run docker-compose up and get expected results, rather than having to supplement every single repository with documentation to remind developers to run a chain of commands or scripts to pull the latest availabe images before starting the apps.

It's not about "the functionality already exists to do this by running these commands", it's a better user experience.

Imagine when someone suggested adding --stop to docker-compose rm that the answer was "what's wrong with docker-compose stop myapp && docker-compose rm myapp, or if someone had requested the implementation of docker-compose down they were just asked why is docker-compose stop myapp && docker-compose rm -v myapp && docker-compose images -q myapp | xargs docker rmi ... impractical?

@jbauson

This comment has been minimized.

jbauson commented Jun 30, 2018

This thread has been raised 2years ago and I still thought that adding a flag in the docker-compose.yml is the best way. In my case, we have like 37 services in a swarm, and updating 4-5 images from that total is not easy. I just created a shell for now that can monitor the change from the repository and do the pull for the specific image before recreating the service if it has been updated.

@jrynyt

This comment has been minimized.

jrynyt commented Jul 10, 2018

Another point here is that docker-compose up has a --quiet-pull option. When I was first trying to figure out if up included a pull, I assumed it did based on its presence. It goes to reason that not using --quiet-pull results in a verbose pull.

@eaglet3d

This comment has been minimized.

eaglet3d commented Jul 19, 2018

Two years of people trying to convince the Docker Compose maintainers that having a --pull option would be a better experience without having to run a separate command. If the docker-compose maintainers just implemented the feature, to begin with, everyone's lives would be better. It seems to be clear that the current maintainers don't want to add this feature for the betterment of the community.

Maybe we should just fork docker-compose and update it ourselves.

@bean5

This comment has been minimized.

bean5 commented Jul 20, 2018

@ktravelet

This comment has been minimized.

ktravelet commented Jul 27, 2018

I ran into this same issue and holy crap, a bunch of whiners in here. This is FREE open source software. Give the maintainers a break. I'm sure they have much more important things to work on than this. If anyone needs this so badly, why don't you open a PR? If the code is clean with minimal risk, I don't see why they wouldn't accept it.

@dvdgsng

This comment has been minimized.

dvdgsng commented Aug 7, 2018

This issue should be re-opened since the discussion lacks a good reason to not implement this request. People are more likely to start working on open issues rather than closed ones.

@orodbhen

This comment has been minimized.

orodbhen commented Aug 7, 2018

The fact that this "closed" issue has remained so active does suggest that it has not been handled well.

Unfortunately, I've found that responses to issue posts on some GitHub repos are not very helpful. The tone is often terse, and suggests that feedback is less than welcome.

Some have pointed out here that this is an open-source project, and (at least most of us) aren't paying customers. However, it's also worth noting that submitting issue reports constitutes a significant time investment, even more so if you participate in the issue's resolution.

A user or developer, having spent time troubleshooting a problem, and found a workaround, will then decide if they want to spend additional time to report it. An unreceptive response from maintainers will likely result in them opting not to bother the next time around.

The software is not really "FREE", in the "free beer" sense. as we're all trying to participate in making it better. Having people willing to test your software and provide feedback is a valuable resource. Those with the requisite programming skills are even willing to contribute code, but they want some kind of indication that their contributions are welcome before spending time on it.

Obviously, comments that simply complain about a problem and demand that it be fixed aren't useful, but the majority aren't doing that, and comments like "it's open source, just fork it" are equally useless.

@jdalegonzalez

This comment has been minimized.

jdalegonzalez commented Sep 21, 2018

@shin- Why was "--build" implemented? Is it different than docker-compose build && docker-compose up? Just trying to understand the philosophical different between --build (which got added) and --pull (which has been deemed redundant). Understanding the thought process might help me remember how things work. AND if the issue is opened, I'm happy to submit a PR. @everybodyelse... is it really the "spirit of opensource" that if you don't like something you fork? I thought the spirit of opensource was "if you don't like something you a) help contribute to the requirements if that's where you are, b) help contribute to the code if you can" and that you only forked if your requirements were very clearly something only you would benefit from. ie I thought we benefit most when we share - but I'm happy to be educated here.

@alonsohki

This comment has been minimized.

alonsohki commented Oct 2, 2018

After two years of insisting, giving good reasons that are ignored and enormous community support towards having this actually implemented, I would say that this feature won't make it just because @shin- doesn't want to. No reason to keep insisting.

@SuperSandro2000

This comment has been minimized.

SuperSandro2000 commented Oct 2, 2018

There is one reason: docker won't refresh the images if one pull fails and there is no good reason not to do it.

@c9s

This comment has been minimized.

c9s commented Oct 14, 2018

I am looking for the kubernetes image pull policy in docker compose, would be great if the "pull" can be used.

@Wenzil

This comment was marked as disruptive content.

Wenzil commented Nov 12, 2018

@shin- Stop being childish. Plenty of good reasons to implement this feature have been mentioned. At least be open to PRs...

@shin-

This comment was marked as off-topic.

Member

shin- commented Nov 12, 2018

You can disagree with me, but I'm disappointed that you would resort to ad hominems, @Wenzil. Our community generally holds itself to a higher standard.

@lig

This comment was marked as off-topic.

lig commented Nov 13, 2018

@shin- Our community mostly thinks the same keeping it unsaid because of the reason you've mentioned. @Wenzil is just being honest enough to say it out loud.
A lot of people I know prefer to not bother and gave it up on trying to convince docker-compose to move toward its users.

@ntwrkguru

This comment was marked as off-topic.

ntwrkguru commented Nov 13, 2018

Many people disagree @shin- for very valid reasons described. At the very least, this should be a service declaration. Stringing bash commands together isn't a particularly good solution for programmatic deployments.

A lot of people I know prefer to not bother and gave it up on trying to convince docker-compose to move toward its users.

This. And it's not just docker-compose. Docker-stack, docker-machine, and docker-cli are all similar.

@lig

This comment was marked as off-topic.

lig commented Nov 14, 2018

@shin-

This comment has been minimized.

Member

shin- commented Nov 14, 2018

Locking as this keeps getting derailed. We'll re-evaluate depending on the fate of docker/cli#1498

@docker docker locked as spam and limited conversation to collaborators Nov 14, 2018

@shin-

This comment has been minimized.

Member

shin- commented Nov 15, 2018

As an update, we've decided internally to look into adding a pull_policy parameter to service definitions. We still have to figure out what the options and exact syntax will be, but we hope it'll satisfy the needs expressed in this thread.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.