bake randomly fails to pass secrets to build when remote context is involved

[slonopotamus]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

1. Create docker-bake.hcl:

target "vs-console" {
  contexts = {
    base = "target:vs"
  }
  output = [
    {
      type = "cacheonly"
    }
  ]
  secret = [
    {
      id = "SECRET2"
    }
  ]
  dockerfile-inline = <<EOF
  FROM base
  RUN --mount=type=secret,id=SECRET2,env=SECRET2 [ -n "$SECRET2" ] && echo $SECRET2 > /secret2
  EOF
}

target "default" {
  contexts = {
    base = "target:vs-console"
    source-prep = "target:source-prep"
  }
  output = [
    {
      type = "cacheonly"
    }
  ]
  dockerfile-inline = <<EOF
  FROM base
  COPY --from=source-prep /secret1 /secret1_copy
  EOF
}

2. Run SECRET1=foo SECRET2=bar docker buildx bake --no-cache --progress plain -f cwd://docker-bake.hcl https://github.com/slonopotamus/bake-secret-bug.git#main multiple times
3. You may need to docker system prune -af to increase chances to trigger the bug

Expected behaviour

Build always succeeds

Actual behaviour

Build randomly either succeeds or fails with

#10 [source-prep stage-0 1/2] RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "****" ] && echo **** > /secret1
#10 ERROR: process "/bin/sh -c [ -n \"$SECRET1\" ] && echo $SECRET1 > /secret1" did not complete successfully: exit code: 1
------
 > [vs stage-0 1/2] RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "****" ] && echo **** > /secret1:
------

Buildx version

0.36.0

Docker info

% docker info
Client: Docker Engine - Community
 Version:    29.7.2
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  agent: Docker AI Agent Runner (Docker Inc.)
    Version:  v1.119.0
    Path:     /Users/marat/.docker/cli-plugins/docker-agent
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.30.0
    Path:     /Users/marat/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.36.0-desktop.1
    Path:     /Users/marat/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.3.1
    Path:     /Users/marat/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     /Users/marat/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.4.3
    Path:     /Users/marat/.docker/cli-plugins/docker-desktop
  dhi: CLI for managing Docker Hardened Images (Docker Inc.)
    Version:  v0.0.7
    Path:     /Users/marat/.docker/cli-plugins/docker-dhi
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     /Users/marat/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/marat/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.43.3
    Path:     /Users/marat/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.2.6
    Path:     /Users/marat/.docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.6.9
    Path:     /Users/marat/.docker/cli-plugins/docker-offload
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.2.0
    Path:     /Users/marat/.docker/cli-plugins/docker-pass
  sandbox: "docker sandbox" is deprecated, use Docker Sandboxes instead (Docker Inc.)
    Version:  v0.13.0
    Path:     /Users/marat/.docker/cli-plugins/docker-sandbox
  scout: Docker Scout (Docker Inc.)
    Version:  v1.24.0
    Path:     /Users/marat/.docker/cli-plugins/docker-scout

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 29.7.2
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e53c7c1516c3b2bff98eb76f1f4117477e6f4e66
 runc version: v1.3.6-0-g491b69ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.75GiB
 Name: docker-desktop
 ID: 4f0e38ed-29f6-45bf-957a-7ec605814357
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/marat/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables

Builders list

% docker buildx ls                           
NAME/NODE           DRIVER/ENDPOINT     STATUS    BUILDKIT   PLATFORMS
default             docker                                   
 \_ default          \_ default         running   v0.32.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.32.2    linux/amd64 (+2), linux/arm64, linux/ppc64le, linux/s390x, (2 more)

Configuration

Remote bake definition that is used in the reproduction:

target "base" {
  output = [
    {
      type = "cacheonly"
    }
  ]
  dockerfile-inline = <<EOF
  FROM busybox
  EOF
}

target "source-prep" {
  contexts = {
    base = "target:base"
  }
  secret = [
    {
      id = "SECRET1"
    }
  ]
  output = [
    {
      type = "cacheonly"
    }
  ]
  dockerfile-inline = <<EOF
  FROM base
  RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "$SECRET1" ] && echo $SECRET1 > /secret1
  EOF
}

target "vs" {
  contexts = {
    base = "target:base"
    source-prep = "target:source-prep"
  }
  output = [
    {
      type = "cacheonly"
    }
  ]
  dockerfile-inline = <<EOF
  FROM base
  COPY --from=source-prep /secret1 /
  EOF
}

Build logs

#0 building with "desktop-linux" instance using docker driver

#1 [internal] load git source https://github.com/slonopotamus/bake-secret-bug.git#main
#1 0.672 327609d671717f96b3a6e57baef7f08b5181a551       refs/heads/main
#1 ...

#2 [internal] load local bake definitions
#2 reading docker-bake.hcl 571B / 571B done
#2 DONE 0.0s

#1 [internal] load git source https://github.com/slonopotamus/bake-secret-bug.git#main
#1 0.691 327609d671717f96b3a6e57baef7f08b5181a551       refs/heads/main
#1 CACHED

#3 [base internal] load build definition from Dockerfile
#3 transferring dockerfile: 52B done
#3 DONE 0.0s

#4 [base internal] load metadata for docker.io/library/busybox:latest
#4 DONE 0.3s

#5 [vs-console internal] load build definition from Dockerfile
#5 transferring dockerfile: 144B done
#5 DONE 0.0s

#6 [default internal] load build definition from Dockerfile
#6 transferring dockerfile: 98B done
#6 DONE 0.0s

#7 [vs internal] load build definition from Dockerfile
#7 transferring dockerfile: 86B done
#7 DONE 0.0s

#8 [source-prep internal] load build definition from Dockerfile
#8 transferring dockerfile: 144B done
#8 DONE 0.0s

#9 [default 1/1] FROM docker.io/library/busybox:latest@sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616
#9 resolve docker.io/library/busybox:latest@sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616 done
#9 CACHED

#10 [vs stage-0 1/2] RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "****" ] && echo **** > /secret1
#10 ERROR: process "/bin/sh -c [ -n \"$SECRET1\" ] && echo $SECRET1 > /secret1" did not complete successfully: exit code: 1
------
 > [default stage-0 1/2] RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "****" ] && echo **** > /secret1:
------
Dockerfile:2
--------------------
   1 |       FROM base
   2 | >>>   RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "$SECRET1" ] && echo $SECRET1 > /secret1
   3 |     
--------------------
Dockerfile:2
--------------------
   1 |       FROM base
   2 | >>>   RUN --mount=type=secret,id=SECRET1,env=SECRET1 [ -n "$SECRET1" ] && echo $SECRET1 > /secret1
   3 |     
--------------------
ERROR: target default: failed to solve: process "/bin/sh -c [ -n \"$SECRET1\" ] && echo $SECRET1 > /secret1" did not complete successfully: exit code: 1

Additional info

1. It is important to use a mix of local and remote Bake file definitions. I cannot reproduce the issue if only local or only remote definition is used.
2. Bug does not reproduce if you build non-default targets in given testcase
3. Bug is more likely to reproduce after docker system prune -af
4. I am using a nontrivial hierarchy of targets where intermediate target uses secrets. Such case is very close to bake secrets are ignored from transitive targets #3056. If you look at that issue, you will notice that it has a comment claiming the issue was not fixed. Nontrivial hierarchy is important for reproduction of current bug, I cannot reproduce it by building directly the target that uses secrets.
5. It is important that both remote and local docker-bake.hcl have a target that uses secrets.
6. In real-world scenario, I am also observing secrets failing to be passed to ADD <git url> and ADD <http url> directives.
7. Bug reproduces for both Linux and Windows containers.