docker-compose build results in repository pull errors for local images

[TrentonAdams]

docker-compose build results in repository pull errors for local images

Sample broken project at: https://github.com/TrentonAdams/compose-bug

According to the docker compose documentation

If you specify image as well as build, then Compose names the built image with the webapp and optional tag specified in image:

It used to do just that, but now it just breaks.

Running this just works...

./build.sh

Running this fails with repository pull errors...

docker-compose build

The project this sample project is based off of has been virtually untouched for a couple of years, but now it's not working the way that it used to. Docker compose seems to have a regression.

Steps to reproduce the issue:

1. docker-compose build

Describe the results you received:

------
 > [compose-play_backend internal] load metadata for docker.io/library/prj-base:latest:
------
------
 > [compose-play_frontend internal] load metadata for docker.io/library/prj-build:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Describe the results you expected:

Completion of the build

Output of docker compose version:

Docker Compose version v2.2.3

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.3)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 241
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.64GiB
 Name: docker-desktop
 ID: ZECY:N6A4:CL2S:P5QZ:LAGL:KJUT:CRYN:ZLIX:YTAL:VIIB:SXYY:JDQN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.254.0.0/16, Size: 24

Additional environment details:

uname -a
Darwin C02ZTEXGMD6T 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro16,1 Darwin

[ndeloof]

It seems build fails while pulling the base image use in your service's Dockerfile.
Can you check a plain good old docker build is successful for your Dockerfiles when Buildkit is enabled?
( run DOCKER_BUILDKIT=1 docker build <context_path>)

[TrentonAdams]

@ndeloof Indeed it does work, as mentioned. Just run ./build.sh

[ndeloof]

Ok got it.
The issue here is that your second service is built using another service image as base image. But buildkit builds are strictly isolated, and can't manage such cross-build dependencies.
see docker/buildx#447

[ndeloof]

it seems this buildx limitation has a workaround: https://github.com/docker/buildx/blob/v0.8.0-rc1/docs/reference/buildx_bake.md#using-a-result-of-one-target-as-a-base-image-in-another-target
but this require some explicit declaration, and as such need to parse Dockerfile... :'(

[ndeloof]

Ran a quick prototype:
we can set build options with Inputs.NamedContexts set to all (other) services NamedContext{ Path: service.Build.Context }. Doing so, buildkit knows about other images being built and will build them in dependency order (and still improve parallelism where feasible)
Unfortunately this fails with unsupported frontend capability moby.buildkit.frontend.contexts if one does not force # syntax=docker/dockerfile-upstream:1.4.0-rc1 in Dockerfile as a recent buildkit version need to be involved engine-side. Nest engine update should help fixing this last limitation.

[TrentonAdams]

Ok got it. The issue here is that your second service is built using another service image as base image. But buildkit builds are strictly isolated, and can't manage such cross-build dependencies. see docker/buildx#447

I'm wondering, why did this previously work then? I mean I originally setup my project to use buildkit features (e.g. --mount), so it would have never worked if buildkit was always doing it the way it is now.

[ndeloof]

docker-compose v1 run builds sequentially, so even with buidlkit enabled, you get base images to have been built when service image build is triggered

[TrentonAdams]

@ndeloof So my impression is that this is being considered an issue to fix then yes?

[ndeloof]

Yes this is an issue we'd like to fix, but the fix is not trivial. Work on progress

[TrentonAdams]

@ndeloof Yep, understood. I can visualize how it might be difficult. Perhaps some sort of centralized cache/query code that determines if something needs building before hand? That's a wild ass guess, I have never looked at the code, lol.

[ndeloof]

duplicates #8538

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[typoworx-de]

Problem is still there. I had these problems too. I figured out later it obviously is related to the problem adressed in this ticket as well. See my ticket here: moby/buildkit#4162