net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity" on docker engine REST API

[h8liu]

What version of Go are you using (go version)?

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/h8liu/Library/Caches/go-build"
GOENV="/Users/h8liu/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/h8liu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/h8liu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/vf/2lnfy_h578s0mc2911f90qc00000gn/T/go-build362761685=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

• Install docker on Mac OS app: https://docs.docker.com/docker-for-mac/install/
• Create a container docker create --name test alpine sleep 1000
• Start it in background docker start test
• go run the following program:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/url"
	"strings"
)

func main() {
	// Before running this needs to:
	// - Have docker 19.03.12 installed
	// - Run "docker create --name test alpine sleep 10000" to create the container
	// - Run "docker start test" to start it.
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	d := new(net.Dialer)
	x := &http.Transport{
		DialContext: func(ctx context.Context, net, addr string) (net.Conn, error) {
			return d.DialContext(ctx, "unix", "/var/run/docker.sock")
		},
	}

	c := &http.Client{Transport: x}

	u := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/containers/test/exec",
	}
	var req = struct {
		Cmd          []string
		AttachStdout bool
		AttachStderr bool
	}{
		Cmd:          []string{"echo", "hello"},
		AttachStderr: true,
		AttachStdout: true,
	}
	body, err := json.Marshal(req)
	if err != nil {
		return err
	}
	header := make(http.Header)
	header.Add("Content-Type", "application/json")
	httpReq := &http.Request{
		Method: "POST",
		URL:    u,
		Header: header,
		Body:   ioutil.NopCloser(bytes.NewBuffer(body)),
	}
	resp, err := c.Do(httpReq)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	bodyBytes, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return err
	}
	var createResp struct {
		ID string `json:"Id"`
	}
	if err := json.Unmarshal(bodyBytes, &createResp); err != nil {
		return err
	}

	u2 := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/exec/" + createResp.ID + "/start",
	}

	httpReq2 := &http.Request{
		Method: "POST",
		URL:    u2,
		Header: header,
		Body:   ioutil.NopCloser(strings.NewReader("{}")),
	}
	resp2, err := c.Do(httpReq2)
	if err != nil {
		return err
	}

	defer resp2.Body.Close()
	bodyBytes2, err := ioutil.ReadAll(resp2.Body)
	if err != nil {
		return err
	}
	fmt.Println(string(bodyBytes2))

	return nil
}

What did you expect to see?

See hello.

What did you see instead?

2020/08/12 10:37:28 Post "http://uds/v1.40/exec/bea64d1e75682a89d1e75d59da45fde21977aa8103b523a5b6e9d8a7eba85ab4/start": net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity"
exit status 1

Docker version

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

---

Some additional notes:

• The issue does not reproduce on go1.14.7 run (because the transfer encoding enforcement was not there I think).
• The issue (weirdly) does not reproduce on a Linux machine of mine. Maybe the docker server is on a different version, and handles requests differently?

[h8liu]

after some learning, this seems to be expected behavior..

[rkuprov]

@h8liu how did you solve it? I'm having a similar issue.

[h8liu]

I did not solve it. I mitigated it by running all stuff on a linux machine. The issue seems to be OSX only. I am guessing that the OSX docker desktop app has some kind of reverse proxy for the socket that is doing stuff incorrectly, but I don't know for sure.

[h8liu]

I closed it because I don't think golang client library is doing anything wrong, although this seems like a regression.

[sazzer]

It's worth mentioning - since I came here from the testcontainers-go issue - that the Docker daemon on macOS works perfectly well with the appropriate TestContainers libraries for Go 1.14.7, with Rust, with Node and with Java, as well as just the standard Docker tooling. It only doesn't work with Go 1.15.

I'm not sure if the Go libraries for communicating with Docker work differently than in all those other languages - given that Docker itself is written in Go, that is possible - but when something is broken in only one version of one language which has itself only just been released, it very much feels like a newly introduced bug.

[iulian-gm]

I also stumbled upon this via testcontainers-go (testcontainers/testcontainers-go#232). Having the exact same issue with Go 1.15 only.

[davecheney]

@sazzer @iulian-gm as you have noted that In your opinion this issue is not fixed would you please raise a new issue, fill out the template and provide a way for someone to reproduce the issues you are experiencing. Thank you

[tankbusta]

I'm seeing the same thing - can this issue just be reopened? It follows the template and provides a way to reproduce

Edit: It might have been caused by the work to fix #38867 but I couldn't find any points in Docker that explicitly set the transfer encoding to identity (although I'm not too familiar with transfer-encoding/identity). Debugging the example provided by @h8liu does indeed show "identity" being returned by the server

[davecheney]

Thank you to those who have commented.

Re new issue vs continuing one that was closed without conclusion. It is my experience that the likelihood of an issue being closed successfully is inversely proportionate to its length.

Focused issues with clear instructions to reproduce the problem are often closed quickly.

To that end, please default to raising new issues. If there is a cost in terms of deduplicating issues that is one myself and the other issue gardeners are happy to invest in to see issues diagnosed and resolved quickly.

[h8liu]

reopened per requests @tankbusta

the head of this thread has a working example program that can reproduce the issue on OSX, but it does not reproduce on Linux (with very close docker versions that are compiled with the same golang version).

So I think it is not that the docker API server socket is broken, but there is an adopter layer specifically used in OSX's docker app, that is forwarding the http requests between the http client and server incorrectly.

At the very least, I think golang's http clients in go1.15 works well with former golang http servers, which is why I think it is not golang's "fault", although it is certainly a regression (breaking change), but for a good security reason.

I am also filing an issue to docker/for-mac: https://github.com/docker/for-mac/issues

[h8liu]

imho, I don't think golang team needs to do anything. I am just leaving this issue open so that people can more easily discover this. the root cause of this issue very likely lies in docker's macos app.