[BUG] compose 2.20.1 broke image building on remote Engine (still not fixed in 2.26.0)

[LaXiS96]

Description

My use case is building/deploying from a Windows client on/to remote Linux Engine instances. I have been using compose 2.16.0 successfully for a long time until I decided to upgrade to the latest 2.23.0.
I have found the breaking point to be version 2.20.1.

Environment variables:

COMPOSE_FILE=C:\Users\<redacted>\AppData\Local\Temp\tmp9EA8.tmp
COMPOSE_PROJECT_NAME=bm
COMPOSE_CONVERT_WINDOWS_PATHS=1
COMPOSE_PARALLEL_LIMIT=1
DOCKER_BUILDKIT=1
DOCKER_REGISTRY=<redacted>.azurecr.io/
DOCKER_HOST=ssh://root@10.42.64.1

Docker info:

PS> docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.10.0-docker)
  compose: Docker Compose (Docker Inc., v2.20.1)

Server:
 Containers: 8
  Running: 8
  Paused: 0
  Stopped: 0
 Images: 56
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.10.162-1.ph4
 Operating System: VMware Photon OS/Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.773GiB
 Name: photon-9662f57d592b
 ID: 37P5:MBLZ:TGUE:HROE:PPZN:56OC:RT7J:ODLZ:B4PZ:URHC:3OGI:EQJJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Not working with 2.20.1:

PS> docker compose build --pull <redacted>
[+] Building 0.0s (0/0)
listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: failed to write client preface: write |1: file already closed"

It takes more than 3 minutes from [+] Building 0.0s (0/0) appearing to the command erroring out (but the remote Engine host is just an Hyper-V VM running PhotonOS, on the same machine through a virtual network switch).
This behavior is consistent from try to try.

Working with 2.20.0 (exact same environment, only difference is the compose exe):
(notice the http2 error: I could only reproduce it once, all other times it was never reported, but it did not break the build in this instance)

PS> docker compose build --pull <redacted>
2023/11/07 13:31:30 http2: server: error reading preface from client dummy-1: read |0: file already closed
[+] Building 1.4s (17/17) FINISHED
 => [<redacted> internal] load build definition from Dockerfile                                                                                                                                                                                              0.2s
 => => transferring dockerfile: 32B                                                                                                                                                                                                                                                0.0s
 => [<redacted> internal] load .dockerignore                                                                                                                                                                                                                 0.2s
 => => transferring context: 2B                                                                                                                                                                                                                                                    0.0s
 => [<redacted> internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                                                                   1.1s
 => [<redacted> internal] load build context                                                                                                                                                                                                                 0.0s
 => => transferring context: 728B                                                                                                                                                                                                                                                  0.0s
 => [<redacted>  1/12] FROM docker.io/library/ubuntu:20.04@sha256:ed4a42283d9943135ed87d4ee34e542f7f5ad9ecf2f244870e23122f703f91c2                                                                                                                           0.0s
 => CACHED [<redacted>  2/12] COPY 99fixbadproxy /etc/apt/apt.conf.d/                                                                                                                                                                                        0.0s
 => CACHED [<redacted>  3/12] COPY ca.pem /usr/local/share/ca-certificates/<redacted>.crt                                                                                                                                                                        0.0s
 => CACHED [<redacted>  4/12] RUN apt-get update &&     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends apt-utils &&     DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends apt-transport-https ca-certifi  0.0s
 => CACHED [<redacted>  5/12] RUN curl -sS https://packages.microsoft.com/keys/microsoft.asc | apt-key add - &&     curl -sS -o /etc/apt/sources.list.d/msprod.list https://packages.microsoft.com/config/ubuntu/20.04/prod.list &&     apt-get update &&    0.0s
 => CACHED [<redacted>  6/12] RUN curl -sS https://packages.elastic.co/GPG-KEY-elasticsearch | apt-key add - &&     echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main' >/etc/apt/sources.list.d/curator.list &&     apt-get   0.0s
 => CACHED [<redacted>  7/12] RUN curl -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc &&     chmod +x /usr/local/bin/mc                                                                                                             0.0s
 => CACHED [<redacted>  8/12] COPY scripts/* /usr/local/bin/                                                                                                                                                                                                 0.0s
 => CACHED [<redacted>  9/12] COPY crontab /etc/cron.d/                                                                                                                                                                                                      0.0s
 => CACHED [<redacted> 10/12] COPY curator/* /root/.curator/                                                                                                                                                                                                 0.0s
 => CACHED [<redacted> 11/12] COPY sqlserver/* /root/.sqlserver/                                                                                                                                                                                             0.0s
 => CACHED [<redacted> 12/12] RUN dos2unix /usr/local/bin/* /etc/cron.d/crontab &&     chmod +x /usr/local/bin/* &&     chmod -R 644 /etc/cron.d/crontab /root/.curator/ /root/.sqlserver/                                                                   0.0s
 => [<redacted>] exporting to image                                                                                                                                                                                                                          0.0s
 => => exporting layers                                                                                                                                                                                                                                                            0.0s
 => => writing image sha256:58fc911450050f9e4c1fdced511c41a3348af897df2714556a8351c91b5e84a0                                                                                                                                                                                       0.0s
 => => naming to <redacted>.azurecr.io/<redacted>

The problem seems related to BuildKit/buildx, I have also tried upgrading compose to 2.23.0, docker CLI to 24.0.6 and the buildx plugin to 0.11.2 but the result is exactly the same.

Steps To Reproduce

No response

Compose Version

No response

Docker Environment

No response

Anything else?

No response

[ndeloof]

Can you successfully build the same image using docker buildx build ... ?

[LaXiS96]

With CLI v20.10.23 and its included buildx v0.10.0-docker, docker buildx build . successfully builds the image.
With CLI v24.0.6 and buildx plugin v0.11.2, it also successfully builds the image.

[ndeloof]

Release v2.23.3 includes an updated version of buildx, would you have a chance to give it a try ?

[LaXiS96]

Thank you for the update, unfortunately the behavior is exactly the same and with the same timings:

[+] Building 0.0s (0/0)                                                                                                                                                                           docker:default
listing workers for Build: failed to list workers: Unavailable: connection error: desc = "transport: failed to write client preface: write |1: file already closed"

(exit code is 17)

Is there any way to enable "debug" mode for compose like I can do for docker CLI (with --debug and --log-level=debug)?

[d4rkmen]

Same expirience here. What is the latest working version?

[LaXiS96]

@d4rkmen I am using 2.20.0 which works

[gpetrov]

Any news on this? Our clients have to downgrade to docker desktop 4.21.0 to be able to build docker images on remotes, very inconvenient.

[Nathanael-Shermett]

I rolled back to 4.20.1 and can confirm the issue does not occur on that version.

[LaXiS96]

@ndeloof any possibility of resolving this issue?
2.26.0 (with Docker CLI 26.0.0) has the exact same behavior, we can't upgrade from 2.20.0 because any newer version breaks building from a Windows CLI to a Linux Engine.

[ndeloof]

I have no idea what's wrong here. Compose fully delegates build to buildx (vendored inside binary) and has no control on the communication between client and (remote) builder

[LaXiS96]

Oh that makes sense.
I tried manually building via buildx 0.13.1 with docker buildx build . and it now behaves exactly like compose does; that's not what I expected, but I probably missed something last time.
I will open an issue in the buildx repository, thank you.

Edit: for anybody following this issue, I have opened docker/buildx#2356
I also tried buildx 0.11.2 again, it does actually work.