[BUG] building w/ compose bake always results in `NotFound desc = no such job xxxxxxxxx` errors

[nocive]

Description

I don't have a minimal reproduction scenario at this point, but we've tried to switch to compose bake as the build engine for our existing docker compose project for a while and we always get the same type of error coming from a random container.

The errors look as follows:

08:43:09  #141 exporting layers 0.4s done
08:43:12  target some-container: failed to receive status: rpc error: code = NotFound desc = no such job xxxxe8yk4gxypivxclixworqh

We've tried all docker compose versions since the feature is available (including 2.37.1 from a day ago) and all of them produce the same results.

Any idea what it could be or how we can help diagnose it?

Steps To Reproduce

1. enable bake as the build engine w/ COMPOSE_BAKE=true
2. build a previously working project w/ docker compose build --with-dependencies <A LIST OF CONTAINERS>
3. see it error out on an arbitrary container with the following error: target some-container: failed to receive status: rpc error: code = NotFound desc = no such job xxxxe8yk4gxypivxclixworqh

Compose Version

2.36.1
2.36.2
2.37.0
2.37.1

Docker Environment

Client: Docker Engine - Community
 Version:    28.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.1
    Path:     /home/xxx/.docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 139
 Server Version: 28.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-31-cloud-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 14.27GiB
 Name: xxxx
 ID: 20e43fc1-f756-441c-aa46-ab1a663b48ae
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: sisubot
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Note: Error also occurred with older 27.x versions.

Anything else?

No response

[ndeloof]

Please run docker compose build <...args> --print and capture output into a bake.json file
Then run docker buildx bake -f bake.json to double check this same error applies when directly running bake.

[nocive]

@ndeloof thanks for the guidance.

And yes, the same error occurs when directly running bake.

[nocive]

With debug enabled, I get the following additional (and potentially useful) output:

10:01:28  5340 v0.24.0 /usr/libexec/docker/cli-plugins/docker-buildx buildx -D bake -f bake.json
10:01:28  github.com/docker/buildx/build.BuildWithResultHandler.func2.5.2
10:01:28  	github.com/docker/buildx/build/build.go:584
10:01:28  github.com/docker/buildx/build.BuildWithResultHandler.func2.5
10:01:28  	github.com/docker/buildx/build/build.go:590
10:01:28  golang.org/x/sync/errgroup.(*Group).Go.func1
10:01:28  	golang.org/x/sync@v0.13.0/errgroup/errgroup.go:79
10:01:28  runtime.goexit
10:01:28  	runtime/asm_amd64.s:1700
10:01:28  
10:01:28  5340 v0.24.0 /usr/libexec/docker/cli-plugins/docker-buildx buildx -D bake -f bake.json
10:01:28  github.com/moby/buildkit/client.(*Client).solve.func4
10:01:28  	github.com/moby/buildkit@v0.22.0/client/solve.go:328
10:01:28  golang.org/x/sync/errgroup.(*Group).Go.func1
10:01:28  	golang.org/x/sync@v0.13.0/errgroup/errgroup.go:79

[ndeloof]

cc @crazy-max

[crazy-max]

Can you give a minimal repro with Dockerfile(s) and Compose file?

Also what's the output of docker info

[nocive]

Can you give a minimal repro with Dockerfile(s) and Compose file?

Not really, not at this point.
We're only able to reproduce this with our project (which I sadly can't share) and in our CI environments.

Also what's the output of docker info

Client: Docker Engine - Community
 Version:    28.2.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.24.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.37.1
    Path:     /home/xxx/.docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 139
 Server Version: 28.2.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-31-cloud-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 14.27GiB
 Name: xxxx
 ID: 20e43fc1-f756-441c-aa46-ab1a663b48ae
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: sisubot
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false