[BUG] Building with Compose on remote Docker server via TLS TCP by setting DOCKER_HOST is broken

[GethDeeo]

Description

After upgrading Docker to version 28.0.4, building with Compose on remote Docker server via TLS encrypted TCP socket is broken if the connection is defined by setting DOCKER_HOST, DOCKER_CERT_PATH and DOCKER_TLS_VERIFY environment variables.

Output:

Error response from daemon: Client sent an HTTP request to an HTTPS server.

Using Docker context with the same values still works.

Steps To Reproduce

1. Create compose.yaml with the following content:

   name: compose
   services:
     foo:
       build:
         dockerfile_inline: FROM alpine
   

2. Run a DinD container in the background acting as remote Docker server:
   docker run -d --rm --name dind --privileged --security-opt='no-new-privileges=false' -p 127.0.0.1:2376:2376/tcp -v ./certs:/certs/client docker:dind
3. Create a Docker context that points to the DinD container:
   docker context create dind --docker "host=tcp://localhost:2376,ca=$(pwd)/certs/ca.pem,cert=$(pwd)/certs/cert.pem,key=$(pwd)/certs/key.pem"
4. Verify that building on remote Docker server by using context works:
   DOCKER_CONTEXT=dind docker build -t foo - <<< 'FROM alpine' && docker exec dind docker image ls
5. Verify that building with Compose on remote Docker server by using context works:
   DOCKER_CONTEXT=dind docker compose build && docker exec dind docker image ls
6. Clean up previously built images in the DinD container:
   docker exec dind docker image rm foo compose-foo
7. Verify that building on remote Docker server by setting DOCKER_HOST works:
   DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 docker build -t foo - <<< 'FROM alpine' && docker exec dind docker image ls
8. Building with Compose on remote Docker server by setting DOCKER_HOST exhibits the bug:
   DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 docker compose build
9. Clean up:
   docker context rm dind && docker container stop dind

Compose Version

Docker Compose version v2.34.0

Docker Environment

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 28.0.4
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: local
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 753481ec61c7c8955a23d6ff7bc8e4daed455734
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
  no-new-privileges
 Kernel Version: 5.15.167.4-microsoft-standard-WSL2
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.59GiB
 Name: geth-desktop
 ID: 5814c356-970f-4d80-8dfa-4bbe4da79f33
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: gethdeeo
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.25.0.0/16, Size: 24
   Base: fd4f:edd1:2b21:1011::/64, Size: 80

Anything else?

No response

[ndeloof]

despite your detailed reproduction scenario (thanks!), I can't reproduce

$  docker compose version
Docker Compose version v2.34.0-desktop.1

$ DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 docker compose build
Compose can now delegate builds to bake for better performance.
 To do so, set COMPOSE_BAKE=true.
[+] Building 0.4s (6/6) FINISHED                                                                                                                                                                                                                                                  docker:default
 => [foo internal] load build definition from Dockerfile                                                                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 85B                                                                                                                                                                                                                                                         0.0s
 => [foo internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                                                                        0.4s
 => [foo internal] load .dockerignore                                                                                                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                             0.0s
 => CACHED [foo 1/1] FROM docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c                                                                                                                                                           0.0s
 => [foo] exporting to image                                                                                                                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                                                                                                                                     0.0s
 => => writing image sha256:17d718589cfc723adc59ddbd5748b9671b7761097ea631ee74d1239009739cda                                                                                                                                                                                                0.0s
 => => naming to docker.io/library/compose-foo                                                                                                                                                                                                                                              0.0s
 => [foo] resolving provenance for metadata file                                                                                                                                                                                                                                            0.0s
[+] Building 1/1
 ✔ foo  Built    

[ndeloof]

note: docker compose relies on docker CLI to manage connexion to remote docker host (same codebase), so until distinct versions are running I can't explain a different behavior

[GethDeeo]

Hi, I gave it another spin. Can you retry it with COMPOSE_BAKE set to true?

I've tried it with docker -D compose build, and that's when I've realized I've turned on Bake support in Compose.

[ndeloof]

$ COMPOSE_BAKE=true DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 docker compose build
[+] Building 0.0s (0/0)                                                                                                                                                                                                                                                                          
Error response from daemon: Client sent an HTTP request to an HTTPS server.

$ DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 docker buildx bake -f compose.yaml 
[+] Building 0.6s (6/6) FINISHED                                                                                                                                                                                                                                                  docker:default
 => [internal] load local bake definitions                                                                                                                                                                                                                                                  0.0s
 => => reading compose.yaml 65B / 65B                                                                                                                                                                                                                                                       0.0s
...

confirmed this is a bug

[ndeloof]

note: running buildx standalone demonstrates the same error:

$ DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 ~/.docker/cli-plugins/docker-buildx bake -f compose.yaml 
ERROR: Error response from daemon: Client sent an HTTP request to an HTTPS server.

@crazy-max can you please confirm running buildx standalone is supported ? Sounds like there's something missing in docker/cli plugin mechanism to setup docker engine API client for cli plugins ran this way

[ndeloof]

I don't have permission to transfert this issue to docker/buildx repository, but with reported reproduction example I can confirm something's wrong with buildx running standalone:

$  $  DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 ~/.docker/cli-plugins/docker-compose up -d
[+] Running 1/1
 ✔ Container toto-foo-1  Started    

$ DOCKER_CERT_PATH="$(pwd)/certs" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://localhost:2376 ~/.docker/cli-plugins/docker-buildx bake -f compose.yaml 
ERROR: Error response from daemon: Client sent an HTTP request to an HTTPS server.

[crazy-max]

Issue transferred, will take look, thanks for reporting.