[BUG] COMPOSE_PARALLEL_LIMIT Is no longer limiting the number of concurrent builds

[fuzzybair]

Description

COMPOSE_PARALLEL_LIMIT was added to restrict the number of concurrent containers being built #9091 (comment). This setting is no longer taking effect resulting in compose attempting to build over 30 containers at once.

Steps To Reproduce

1. Environment: Docker Desktop on windows.
2. I don't have a compose i can share but any compose with more images than set in COMPOSE_PARALLEL_LIMIT should be sufficient.
3. run docker compose build
4. compose will start building more images than allowed.

Compose Version

Docker Compose version v2.38.2-desktop.1

Docker Environment

Docker Desktop 4.43.1 (198352) 

Anything else?

No response

[leoperegrino]

can't reproduce this using:

docker compose version
# Docker Compose version 2.38.2

# compose.yaml
services:
  test_1:
    build:
      dockerfile_inline: |
        FROM alpine
        RUN echo 1 && sleep 10
  test_2:
    build:
      dockerfile_inline: |
        FROM alpine
        RUN echo 2 && sleep 10
  test_3:
    build:
      dockerfile_inline: |
        FROM alpine
        RUN echo 3 && sleep 10
  test_4:
    build:
      dockerfile_inline: |
        FROM alpine
        RUN echo 4 && sleep 10

running COMPOSE_BAKE=false COMPOSE_PARALLEL_LIMIT=1 docker compose build --no-cache builds one image at time as expected.

[ndeloof]

Docker Compose starting with v2.38 delegates the build process to Buildx Bake. The latter doesn't offer such a concurrency limitation mechanism, feel free to request guidance on github.com/docker/buildx

[glours]

If you want to keep the previous behaviour you could use the following command COMPOSE_BAKE=false COMPOSE_PARALLEL_LIMIT=1 docker compose build --no-cache

[ndeloof]

note: COMPOSE_BAKE=false will work for the v2.38.x release but will be removed in a future release, as our plan is to fully rely on Buildx bake as the single docker builder implementation (to avoid code duplication with distinct behaviors)

[dnechay]

@ndeloof is there any planned or suggested workaround on how to control concurrency in a native way for cases when you need to run build/up command for many containers and it exceeds RAM of the machine?

It's not always possible to increase RAM (e.g. if it's developer laptop), and even though we can use something like docker compose config --services | xargs -n 2 docker compose up -d - having concurrency control in a native way is much more cleaner

[ndeloof]

@dnechay you can limit concurrent operation by the builder: https://docs.docker.com/build/buildkit/configure/#max-parallelism

[fuzzybair]

@ndeloof I do not see how bake can fix this. Does compose send the list of containers to bake in a single call? if compose loops over each container and spawns 40 bake processes how is bake expected to limit the number of builds to 10? max-parallelism as it did before applies only to the number of concurrent build threads in a single container.

[ndeloof]

Does compose send the list of containers to bake in a single call

Yes, compose generates a bake config (see docker compose build --print) then bake manages the build including optimizing cross-services dependencies

[fuzzybair]

@ndeloof Thanks for the update. There does not seem to be a way to edit the configuration of the default docker or desktop-linux builders in Docker Desktop but you can add a custom builder with max-parallelism set and I confirmed it does limit the number of parallel processes.