[BUG] docker compose build --pull fails when building an image from another service

[davidrios]

Description

When using --pull option to docker compose build it fails by incorrectly trying to download an image that's referenced from another service.

Steps To Reproduce

Example compose.yaml:

services:
 x_base_service:
  build:
    context: .
    dockerfile_inline: |
      FROM alpine
      RUN echo aaaaaa
 my-service:
  build:
    context: .
    dockerfile_inline: |
      FROM x_base_service
      RUN echo bbbbbbb
    additional_contexts:
      - x_base_service=service:x_base_service

Failing command:

$ docker compose build --pull --no-cache
Compose can now delegate builds to bake for better performance.
 To do so, set COMPOSE_BAKE=true.
[+] Building 2.6s (9/9) FINISHED                                                                                                                                                                                               docker:default
 => [x_base_service internal] load build definition from Dockerfile                                                                                                                                                                      0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                                                      0.0s
 => [x_base_service internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                          1.1s
 => [x_base_service internal] load .dockerignore                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => CACHED [x_base_service 1/2] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c                                                                                             0.0s
 => [x_base_service 2/2] RUN echo aaaaaa                                                                                                                                                                                                 0.3s
 => [x_base_service] exporting to image                                                                                                                                                                                                  0.2s
 => => exporting layers                                                                                                                                                                                                                  0.2s
 => => writing image sha256:eee1685e90cd637930006f6d27838ac5b97eb3ed1f9ac0d50eeb8e0e7b72f370                                                                                                                                             0.0s
 => => naming to docker.io/library/testdockeradd-x_base_service                                                                                                                                                                          0.0s
 => [x_base_service] resolving provenance for metadata file                                                                                                                                                                              0.0s
 => [my-service internal] load build definition from Dockerfile                                                                                                                                                                          0.0s
 => => transferring dockerfile: 74B                                                                                                                                                                                                      0.0s
 => ERROR [my-service context x_base_service] load metadata for testdockeradd-x_base_service                                                                                                                                             0.8s
------
 > [my-service context x_base_service] load metadata for testdockeradd-x_base_service:
------
failed to solve: failed to resolve source metadata for docker.io/library/testdockeradd-x_base_service:latest: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

If not using --pull it builds fine:

$ docker compose build --no-cache
Compose can now delegate builds to bake for better performance.
 To do so, set COMPOSE_BAKE=true.
[+] Building 1.4s (14/14) FINISHED                                                                                                                                                                                             docker:default
 => [x_base_service internal] load build definition from Dockerfile                                                                                                                                                                      0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                                                      0.0s
 => [x_base_service internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                          0.0s
 => [x_base_service internal] load .dockerignore                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => CACHED [x_base_service 1/2] FROM docker.io/library/alpine:latest                                                                                                                                                                     0.0s
 => [x_base_service 2/2] RUN echo aaaaaa                                                                                                                                                                                                 0.3s
 => [x_base_service] exporting to image                                                                                                                                                                                                  0.1s
 => => exporting layers                                                                                                                                                                                                                  0.1s
 => => writing image sha256:72102fc6e03d3f071e951339339b1f9bbbbfe0720972c7976f2510207d7d97cd                                                                                                                                             0.0s
 => => naming to docker.io/library/testdockeradd-x_base_service                                                                                                                                                                          0.0s
 => [x_base_service] resolving provenance for metadata file                                                                                                                                                                              0.0s
 => [my-service internal] load build definition from Dockerfile                                                                                                                                                                          0.0s
 => => transferring dockerfile: 74B                                                                                                                                                                                                      0.0s
 => [my-service context x_base_service] load metadata for testdockeradd-x_base_service                                                                                                                                                   0.0s
 => [my-service internal] load .dockerignore                                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => CACHED [my-service context x_base_service] testdockeradd-x_base_service                                                                                                                                                              0.0s
 => [my-service 1/2] RUN echo bbbbbbb                                                                                                                                                                                                    0.2s
 => [my-service] exporting to image                                                                                                                                                                                                      0.7s
 => => exporting layers                                                                                                                                                                                                                  0.7s
 => => writing image sha256:e1745090a8766e82ad70034d4fb404fbbf3f1318aadf34dc6ee5c2268543b2d6                                                                                                                                             0.0s
 => => naming to docker.io/library/testdockeradd-my-service                                                                                                                                                                              0.0s
 => [my-service] resolving provenance for metadata file                                                                                                                                                                                  0.0s
[+] Building 2/2
 ✔ my-service      Built                                                                                                                                                                                                                 0.0s
 ✔ x_base_service  Built

Using COMPOSE_BAKE=true also works fine with --pull:

$ COMPOSE_BAKE=true docker compose build --pull --no-cache
[+] Building 1.9s (12/12) FINISHED
 => [internal] load local bake definitions                                                                                                                                                                                               0.0s
 => => reading from stdin 831B                                                                                                                                                                                                           0.0s
 => [x_base_service internal] load build definition from Dockerfile                                                                                                                                                                      0.0s
 => => transferring dockerfile: 65B                                                                                                                                                                                                      0.0s
 => [x_base_service internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                          0.6s
 => [my-service internal] load .dockerignore                                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                                                          0.0s
 => [my-service internal] load build definition from Dockerfile                                                                                                                                                                          0.0s
 => => transferring dockerfile: 74B                                                                                                                                                                                                      0.0s
 => CACHED [x_base_service 1/2] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c                                                                                             0.0s
 => [x_base_service 2/2] RUN echo aaaaaa                                                                                                                                                                                                 0.2s
 => [my-service 1/2] RUN echo bbbbbbb                                                                                                                                                                                                    0.1s
 => [x_base_service] exporting to image                                                                                                                                                                                                  0.1s
 => => exporting layers                                                                                                                                                                                                                  0.1s
 => => writing image sha256:a41fefa21baafb4bff89fe25a2a5b9e52be2b28876313030e6121a6eaf989c20                                                                                                                                             0.0s
 => => naming to docker.io/library/testdockeradd-x_base_service                                                                                                                                                                          0.0s
 => [my-service] exporting to image                                                                                                                                                                                                      1.0s
 => => exporting layers                                                                                                                                                                                                                  1.0s
 => => writing image sha256:bc0a5a3224da94845cd6aa6b8a8c0584204e12c359dd799b89d9091ae6d77432                                                                                                                                             0.0s
 => => naming to docker.io/library/testdockeradd-my-service                                                                                                                                                                              0.0s
 => [x_base_service] resolving provenance for metadata file                                                                                                                                                                              0.0s
 => [my-service] resolving provenance for metadata file                                                                                                                                                                                  0.0s
[+] Building 2/2
 ✔ testdockeradd-my-service      Built                                                                                                                                                                                                   0.0s
 ✔ testdockeradd-x_base_service  Built

Compose Version

$ docker compose version
Docker Compose version 2.36.2
$ docker-compose version
Docker Compose version 2.36.2

Docker Environment

$ docker info
Client:
 Version:    28.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.24.0
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.36.2
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 94
  Running: 4
  Paused: 0
  Stopped: 90
 Images: 45
 Server Version: 28.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 061792f0ecf3684fb30a3a0eb006799b8c6638a7.m
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.14.6-arch1-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.26GiB
 Name: nauvis-arch
 ID: f8dd068c-f6ec-4e1a-bd84-9d7b706d43c4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:32000
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.130.0.0/16, Size: 24
   Base: 10.131.0.0/16, Size: 24

Anything else?

No response

[ndeloof]

support for cross-service build in Docker Compose is a bit hack-ish and has limited features, so the introduction of COMPOSE_BAKE to get better support for such use-cases. In such circumstances, Docker Compose builds service image in dependency order to "mimic" the bake behavior, as x_base_service image is available in local image store after a build and then my-service build won't fail. But with --pull option you ask each (isolated) builds to pull base image, and Compose has no way to detect the external_context is actually used to define your base image.

I guess we will have to close this issue as "not planned", considering the next step for Docker Compose is to make COMPOSE_BAKE=true the default behavior

[davidrios]

It does work fine with bake, so that makes sense. Feel free to close this whatever way you want. Thank you for the support!

[ndeloof]

Closing as we made Bake the default builder in v2.37