Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity" on docker engine REST API #40735

Closed
h8liu opened this issue Aug 12, 2020 · 13 comments

Comments

@h8liu
Copy link

@h8liu h8liu commented Aug 12, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/h8liu/Library/Caches/go-build"
GOENV="/Users/h8liu/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/h8liu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/h8liu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/vf/2lnfy_h578s0mc2911f90qc00000gn/T/go-build362761685=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/url"
	"strings"
)

func main() {
	// Before running this needs to:
	// - Have docker 19.03.12 installed
	// - Run "docker create --name test alpine sleep 10000" to create the container
	// - Run "docker start test" to start it.
	if err := run(); err != nil {
		log.Fatal(err)
	}
}

func run() error {
	d := new(net.Dialer)
	x := &http.Transport{
		DialContext: func(ctx context.Context, net, addr string) (net.Conn, error) {
			return d.DialContext(ctx, "unix", "/var/run/docker.sock")
		},
	}

	c := &http.Client{Transport: x}

	u := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/containers/test/exec",
	}
	var req = struct {
		Cmd          []string
		AttachStdout bool
		AttachStderr bool
	}{
		Cmd:          []string{"echo", "hello"},
		AttachStderr: true,
		AttachStdout: true,
	}
	body, err := json.Marshal(req)
	if err != nil {
		return err
	}
	header := make(http.Header)
	header.Add("Content-Type", "application/json")
	httpReq := &http.Request{
		Method: "POST",
		URL:    u,
		Header: header,
		Body:   ioutil.NopCloser(bytes.NewBuffer(body)),
	}
	resp, err := c.Do(httpReq)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	bodyBytes, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		return err
	}
	var createResp struct {
		ID string `json:"Id"`
	}
	if err := json.Unmarshal(bodyBytes, &createResp); err != nil {
		return err
	}

	u2 := &url.URL{
		Scheme: "http",
		Host:   "uds",
		Path:   "/v1.40/exec/" + createResp.ID + "/start",
	}

	httpReq2 := &http.Request{
		Method: "POST",
		URL:    u2,
		Header: header,
		Body:   ioutil.NopCloser(strings.NewReader("{}")),
	}
	resp2, err := c.Do(httpReq2)
	if err != nil {
		return err
	}

	defer resp2.Body.Close()
	bodyBytes2, err := ioutil.ReadAll(resp2.Body)
	if err != nil {
		return err
	}
	fmt.Println(string(bodyBytes2))

	return nil
}

What did you expect to see?

See hello.

What did you see instead?

2020/08/12 10:37:28 Post "http://uds/v1.40/exec/bea64d1e75682a89d1e75d59da45fde21977aa8103b523a5b6e9d8a7eba85ab4/start": net/http: HTTP/1.x transport connection broken: unsupported transfer encoding: "identity"
exit status 1

Docker version

Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Some additional notes:

  • The issue does not reproduce on go1.14.7 run (because the transfer encoding enforcement was not there I think).
  • The issue (weirdly) does not reproduce on a Linux machine of mine. Maybe the docker server is on a different version, and handles requests differently?
@h8liu
Copy link
Author

@h8liu h8liu commented Aug 12, 2020

after some learning, this seems to be expected behavior..

@h8liu h8liu closed this Aug 12, 2020
@rkuprov
Copy link

@rkuprov rkuprov commented Aug 14, 2020

@h8liu how did you solve it? I'm having a similar issue.

@h8liu
Copy link
Author

@h8liu h8liu commented Aug 15, 2020

I did not solve it. I mitigated it by running all stuff on a linux machine. The issue seems to be OSX only. I am guessing that the OSX docker desktop app has some kind of reverse proxy for the socket that is doing stuff incorrectly, but I don't know for sure.

@h8liu
Copy link
Author

@h8liu h8liu commented Aug 15, 2020

I closed it because I don't think golang client library is doing anything wrong, although this seems like a regression.

@sazzer
Copy link

@sazzer sazzer commented Aug 18, 2020

It's worth mentioning - since I came here from the testcontainers-go issue - that the Docker daemon on macOS works perfectly well with the appropriate TestContainers libraries for Go 1.14.7, with Rust, with Node and with Java, as well as just the standard Docker tooling. It only doesn't work with Go 1.15.

I'm not sure if the Go libraries for communicating with Docker work differently than in all those other languages - given that Docker itself is written in Go, that is possible - but when something is broken in only one version of one language which has itself only just been released, it very much feels like a newly introduced bug.

@iulian-gm
Copy link

@iulian-gm iulian-gm commented Aug 18, 2020

I also stumbled upon this via testcontainers-go (testcontainers/testcontainers-go#232). Having the exact same issue with Go 1.15 only.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Aug 19, 2020

@sazzer @iulian-gm as you have noted that In your opinion this issue is not fixed would you please raise a new issue, fill out the template and provide a way for someone to reproduce the issues you are experiencing. Thank you

@tankbusta
Copy link

@tankbusta tankbusta commented Aug 20, 2020

I'm seeing the same thing - can this issue just be reopened? It follows the template and provides a way to reproduce

Edit: It might have been caused by the work to fix #38867 but I couldn't find any points in Docker that explicitly set the transfer encoding to identity (although I'm not too familiar with transfer-encoding/identity). Debugging the example provided by @h8liu does indeed show "identity" being returned by the server

Screen Shot 2020-08-20 at 09 59 21

@davecheney
Copy link
Contributor

@davecheney davecheney commented Aug 21, 2020

Thank you to those who have commented.

Re new issue vs continuing one that was closed without conclusion. It is my experience that the likelihood of an issue being closed successfully is inversely proportionate to its length.

Focused issues with clear instructions to reproduce the problem are often closed quickly.

To that end, please default to raising new issues. If there is a cost in terms of deduplicating issues that is one myself and the other issue gardeners are happy to invest in to see issues diagnosed and resolved quickly.

@h8liu h8liu reopened this Aug 21, 2020
@h8liu
Copy link
Author

@h8liu h8liu commented Aug 21, 2020

reopened per requests @tankbusta

the head of this thread has a working example program that can reproduce the issue on OSX, but it does not reproduce on Linux (with very close docker versions that are compiled with the same golang version).

So I think it is not that the docker API server socket is broken, but there is an adopter layer specifically used in OSX's docker app, that is forwarding the http requests between the http client and server incorrectly.

At the very least, I think golang's http clients in go1.15 works well with former golang http servers, which is why I think it is not golang's "fault", although it is certainly a regression (breaking change), but for a good security reason.

I am also filing an issue to docker/for-mac: https://github.com/docker/for-mac/issues

@h8liu
Copy link
Author

@h8liu h8liu commented Aug 21, 2020

imho, I don't think golang team needs to do anything. I am just leaving this issue open so that people can more easily discover this. the root cause of this issue very likely lies in docker's macos app.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Aug 21, 2020

@h8liu i would prefer if this issue were closed and discussion happened elsewhere until it was determined that there was a bug in the net/http package.

To be clear, if its a bug in the http package, it'll be fixed. If its related to how another project is using the http package, then that discussion belongs elsewhere.

@h8liu
Copy link
Author

@h8liu h8liu commented Aug 21, 2020

okay, you are the boss. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.