Don't build the same thing twice

[kojiromike]

Even though Docker caches layers and the second build is fast, it's not necessary for Compose to build the same thing twice. As far as I can tell, if two services in the yaml have exactly the same string, you can be sure they will be exactly the same build:

21:02 ~/docker/test $ cat fig.yml 
svc1:
    build: oog
svc2:
    build: oog
21:02 ~/docker/test $ fig build
Building svc2...
 ---> e72ac664f4f0
Successfully built e72ac664f4f0
Building svc1...
 ---> e72ac664f4f0
Successfully built e72ac664f4f0

Naturally, the image built is the same for both services.

There's also the question of the race condition if someone edits the Dockerfile while fig is already building. This is a bit contrived, but does work:

$ cat fig.yml
svc1:
    build: oog
svc2:
    build: oog
$ diff oog/Dockerfile[12]
3c3
<  && echo hi

---
>  && echo bye
$ ( cd oog;ln -sf Dockerfile{1,} ); fig build --no-cache & ( cd oog;sleep 2;ln -sf Dockerfile{2,} ); fg
[1] 1374
Building svc2...
 ---> e72ac664f4f0
Step 1 : RUN sleep 5  && echo hi
 ---> Running in dd0c813779ec
fig build --no-cache
hi
 ---> 9f1aed6d839c
Removing intermediate container dd0c813779ec
Successfully built 9f1aed6d839c
Building svc1...
 ---> e72ac664f4f0
Step 1 : RUN sleep 5  && echo bye
 ---> Running in 3c7d54f6145a
bye
 ---> 02ece1cf8f17
Removing intermediate container 3c7d54f6145a
Successfully built 02ece1cf8f17

Contrived or not, if it didn't attempt to run the same build twice, it wouldn't have divergent images.

[bonndan]

+1

[pikeas]

+1

[hectorj]

+1

[mark-adams]

+1

[tristan0x]

+1

[eidge]

+1

[gnarea]

+1

[hobofan]

Duplicate of #610?

[dansondergaard]

+1

[theotheo]

+1

[petyunchik]

+1

[etcinit]

+1

[etcinit]

I'm trying to use compose to launch multiple containers from the same image but with a different command. Running docker-compose up causes the same image to be built a couple of times, which can take a long time depending on the number of containers I'm launching.

Right now, my workaround is to build and tag the image separately, and then reference it from the compose configuration file.
However, this requires other developers on my team to be aware that just running docker-compose up does not work unless you have previously built and tagged the image.

I think this could be addressed in compose if there was a way to reference built images within the docker-compose.yml file.

[dnephin]

As of Compose 1.6.0 you can use both image and build together to do just that. You can optionally build the image outside of compose, and up will just use the image you built, but if someone else runs build, it will do the right thing and build the same tag.

[etcinit]

Thanks, just saw this on the documentation: https://github.com/docker/compose/blob/master/docs/compose-file.md#build

It definitely solves the problem for me.

[greenboxal]

This solves most of the problem, still, when building images in a shared context(e.g.: a CI server), the images would end having the same name and would override each other.

This isn't an issue as we can just change the project name, changing the prefix of all created objects. The problem is that if you use image with build, the project prefix isn't used.

This can be solved either by exposing by some way the project prefix or if the image from one service could be referenced from other project. The first one can be addressed with a simple shell script but I think that docker-compose should handle this natively, preferably with the second option, as it solves side effects like docker-compose pull attempting to pull the images for the other services referencing the image by its name.

[emilsoman]

@etcinit The link to the doc doesn't work anymore. Can you share how you solved your problem?

[Glideh]

Here is the current link for docker-compose file reference build

[frnco]

What about using partial images?

Generate an image with all common factors and build on top of it.

This may also be resolved together with #1896, though I believe we actually need actually more decoupling of building and running (Only-build/partial images, which would allow devs to use "FROM" to build on top of it), while also allowing for granular control during run-time, which in the end could amount to 2 separate, albeit somewhat related, issues. Depends a lot on the design.

My suggestion would be for the Compose-File to allow two new keys, images and commands. Relating to this issue, images would only build and serve as basis for the services. The key commands would be related to #1896.

[ekkis]

@frnco I agree that greater decoupling between building and running is needed. frankly, if all I specify is a build request, I don't see why docker-compose should think it needs to run anything

[mrname]

Perhaps I am not understanding the solution proposed by @dnephin. I have a compose file like:

service1:
  build: .
  image: my-app:latest
service2:
  build: .
  image: my-app:latest

And yet, when I run docker-compose build service1 and then docker-compose build service2, it is built each time. Would the de-duplication of building only occur if I run docker-compose build?

EDIT: I can only reproduce this using Docker for Mac, so it does not seem to be compose related.

[jbhannah]

@mrname I accomplished this by omitting build from "descendant" services. e.g. for a Rails app with Sidekiq:

version: "3"
service:
  web:
    build: .
    …
  sidekiq:
    image: myproject_web:latest
    …

What @dnephin described with your Compose file would be docker-compose build service1 followed by docker-compose up service2, which should build the image once and use the already-built image to start a container.

[rafaelpivato]

@jbhannah omitting build might present issues when you run docker-compose pull

[zsyh]

just write a separate docker-compose file specially for build
for example:

21:02 ~/docker/test $ cat fig.yml 
svc1:
    image: xxxxx
svc2:
    image: xxxxx

21:02 ~/docker/test $ cat fig-build.yml 
svc1:
    image: xxxxx
    build: oog

21:02 ~/docker/test $ docker-compose -f fig-build.yml build
...........

21:02 ~/docker/test $ docker-compose -f fig.yml up

[elquimista]

I recommend using Procfile with overmind / hivemind / foreman

[bnopacheco]

+1

[mo-mughrabi]

+1

[tflori]

I would prefer a solution that is more natural: when two services define the same context and docker file it should automatically use the same image - the image name could be the first service defined the docker file or {project}_{context}..

To be honest: I don't think someone will work on a 4 year old issue. My hope in the docker community is long gone.

[ekkis]

@tflori I feel the same. I gave up on Docker a while back and have been building serverless on Now

[pmatv]

I've managed this issue in the following way:

version: "3"
services:
  build:
    image: single_image_app
    build:
      context: .
      dockerfile: test-Dockerfile
  svc1:
    image: single_image_app
    depends_on:
      - build
  svc2:
    image: single_image_app
    depends_on:
      - build

[ndeloof]

Both docker-compose in latest releases and Compose V2 use BuildKit as a builder backend, and this one manage building images in parallel with deduplication for equivalent Dockerfiles in compose services.

[roma-glushko]

The solution with a separate build service doesn't work for me. It doesn't seem like docker ever tries to wait for the dependent services. If stackoverflow is true, then depends_on only makes sure that the dependent services will run after their dependencies:

https://stackoverflow.com/a/52699978

[ndeloof]

Yes indeed, depends_on was not designed to apply at build time. There a more general issue here with Dockerfiles which depend on another service image as base image. Still no fix yet

[terion-name]

doesn't work now =(

[jimmygchen]

I've managed this issue in the following way:

version: "3"
services:
  build:
    image: single_image_app
    build:
      context: .
      dockerfile: test-Dockerfile
  svc1:
    image: single_image_app
    depends_on:
      - build
  svc2:
    image: single_image_app
    depends_on:
      - build

I'm on compose v2.12.1, for some reasons I had to add pull_policy: never to all services, so that they don't try to pull from docker hub.