Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Transport hangs when io.Pipe is used as request body #29246

Open
tehmoon opened this Issue Dec 14, 2018 · 0 comments

Comments

Projects
None yet
2 participants
@tehmoon
Copy link

commented Dec 14, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

I just tried with 1.11.3 latest brew and still happening

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/moon/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/moon/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pm/vc42c0nn7cl522r_gsg0njlc0000gn/T/go-build143101879=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Compiled and ran the following program: https://play.golang.org/p/zWXnswPCqmm
This is an HTTP client, which can be tested against a server:

nc -nlvp 8081

What did you expect to see?

If I run the program then wait 1-2 seconds, and start typing lines in my terminal, I'm expecting those same line to be sent to the server in http transport format.

What did you see instead?

What I saw was that if I do wait couple seconds before typing, the http client hangs and nothing is sent to the server.
I did some tracing with tcpdump to confirm that no packet were sent to the server.
On the other end, if I do start typing before the web client sends the http header -- meaning that you usually have the compiling time with go run to fill out stdin with some crap -- the web client does not hang and sends everything correctly.

My own workaround is to make sure data is written to the *io.Writer part before invoking http.Client.Do(req) but I have no idea why this is happening. Perhaps it's a bug, or perhaps it's a feature.

Digging through the stack trace, I stumbled upon a comment:

// probeRequestBody reads a byte from t.Body to see whether it's empty
// (returns io.EOF right away).
//
// But because we've had problems with this blocking users in the past
// (issue 17480) when the body is a pipe (perhaps waiting on the response
// headers before the pipe is fed data), we need to be careful and bound how
// long we wait for it. This delay will only affect users if all the following
// are true:
//   * the request body blocks
//   * the content length is not set (or set to -1)
//   * the method doesn't usually have a body (GET, HEAD, DELETE, ...)
//   * there is no transfer-encoding=chunked already set.
// In other words, this delay will not normally affect anybody, and there
// are workarounds if it does.
func (t *transferWriter) probeRequestBody() {

Which brought me to the issue and @bradfitz talking about io.Pipe. However I did test with req.ContentLength = 1 and it did not do the work.

It seems to hit this part of the code:

	timer := time.NewTimer(200 * time.Millisecond)
	select {
	case rres := <-t.ByteReadCh:
		timer.Stop()
		if rres.n == 0 && rres.err == io.EOF {
			// It was empty.
			t.Body = nil
			t.ContentLength = 0
		} else if rres.n == 1 {
			if rres.err != nil {
				t.Body = io.MultiReader(&byteReader{b: rres.b}, errorReader{rres.err})
			} else {
				t.Body = io.MultiReader(&byteReader{b: rres.b}, t.Body)
			}
		} else if rres.err != nil {
			t.Body = errorReader{rres.err}
		}
	case <-timer.C:
		// Too slow. Don't wait. Read it later, and keep
		// assuming that this is ContentLength == -1
		// (unknown), which means we'll send a
		// "Transfer-Encoding: chunked" header.
		t.Body = io.MultiReader(finishAsyncByteRead{t}, t.Body)
		// Request that Request.Write flush the headers to the
		// network before writing the body, since our body may not
		// become readable until it's seen the response headers.
		t.FlushHeaders = true
	}

The case <- timer.C does not make sense to me, I don't know the library really deep and I am trying to understand, but my sense is that it feels like a bug.

Perhaps it is not, but I would be really grateful if somebody explains why this is happening and more importantly, if it is the expected behavior or not.

PC=0x7fff5fe401b2 m=0 sigcode=0

goroutine 0 [idle]:
runtime.pthread_cond_wait(0x145a1a0, 0x145a160, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/sys_darwin.go:302 +0x51
runtime.semasleep(0xffffffffffffffff, 0x10493d1)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/os_darwin.go:63 +0x85
runtime.notesleep(0x1459f60)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/lock_sema.go:167 +0xe3
runtime.stopm()
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/proc.go:2016 +0xe3
runtime.findrunnable(0xc000024500, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/proc.go:2487 +0x4dc
runtime.schedule()
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/proc.go:2613 +0x13a
runtime.park_m(0xc000001500)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/proc.go:2676 +0xae
runtime.mcall(0x10556ab)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/asm_amd64.s:299 +0x5b

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0000164b4)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0000164b4)
        /usr/local/Cellar/go/1.11.2/libexec/src/sync/waitgroup.go:130 +0x64
main.main()
        /tmp/blah/client/main.go:46 +0x237

goroutine 4 [select]:
io.(*pipe).Write(0xc0000a6140, 0xc000100001, 0x4, 0xfff, 0x1, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/io/pipe.go:87 +0x1cc
io.(*PipeWriter).Write(0xc00000e038, 0xc000100000, 0x5, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/io/pipe.go:153 +0x4c
main.main.func1(0xc00000e038, 0xc0000164b4)
        /tmp/blah/client/main.go:30 +0xb5
created by main.main
        /tmp/blah/client/main.go:22 +0x1f7

goroutine 5 [select]:
net/http.(*persistConn).roundTrip(0xc000110000, 0xc000094e40, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:2101 +0x56a
net/http.(*Transport).roundTrip(0x1454620, 0xc0000fe000, 0x203000, 0xc000055d48, 0x11d511a)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:465 +0x9b1
net/http.(*Transport).RoundTrip(0x1454620, 0xc0000fe000, 0x1454620, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0000fe000, 0x12bef80, 0x1454620, 0x0, 0x0, 0x0, 0xc00000e058, 0x0, 0xc000055dd8, 0x1)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc000094cf0, 0xc0000fe000, 0x0, 0x0, 0x0, 0xc00000e058, 0x0, 0x1, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc000094cf0, 0xc0000fe000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc000094cf0, 0xc0000fe000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/client.go:509 +0x35
main.main.func2(0xc0000fe000, 0xc0000164b4)
        /tmp/blah/client/main.go:39 +0x45
created by main.main
        /tmp/blah/client/main.go:37 +0x226

goroutine 37 [IO wait]:
internal/poll.runtime_pollWait(0x2400ea0, 0x72, 0xc000055a88)
        /usr/local/Cellar/go/1.11.2/libexec/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00013a018, 0x72, 0xffffffffffffff00, 0x12bfac0, 0x14243e8)
        /usr/local/Cellar/go/1.11.2/libexec/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00013a018, 0xc000154000, 0x1000, 0x1000)
        /usr/local/Cellar/go/1.11.2/libexec/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00013a000, 0xc000154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc00013a000, 0xc000154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000114018, 0xc000154000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/net.go:177 +0x68
net/http.(*persistConn).Read(0xc000110000, 0xc000154000, 0x1000, 0x1000, 0x1006493, 0xc000131470, 0x102a352)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:1497 +0x75
bufio.(*Reader).fill(0xc000152000)
        /usr/local/Cellar/go/1.11.2/libexec/src/bufio/bufio.go:100 +0x106
bufio.(*Reader).Peek(0xc000152000, 0x1, 0xc0000ae2a0, 0xc000131558, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/bufio/bufio.go:132 +0x3f
net/http.(*persistConn).readLoop(0xc000110000)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:1338 +0x941

goroutine 38 [chan receive]:
net/http.finishAsyncByteRead.Read(0xc000162000, 0xc000170000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transfer.go:1030 +0x60
io.(*multiReader).Read(0xc000164020, 0xc000170000, 0x8000, 0x8000, 0x3, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/io/multi.go:26 +0xaf
net/http.transferBodyReader.Read(0xc000162000, 0xc000170000, 0x8000, 0x8000, 0x1, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transfer.go:62 +0x56
io.copyBuffer(0x2444040, 0xc000166020, 0x12bf9e0, 0xc000162000, 0xc000170000, 0x8000, 0x8000, 0x123aae0, 0x1456a00, 0x2444040)
        /usr/local/Cellar/go/1.11.2/libexec/src/io/io.go:402 +0x125
io.Copy(0x2444040, 0xc000166020, 0x12bf9e0, 0xc000162000, 0xc000166020, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/io/io.go:364 +0x5a
net/http.(*transferWriter).writeBody(0xc000162000, 0x12bed40, 0xc000134040, 0x2, 0x2)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transfer.go:355 +0x155
net/http.(*Request).write(0xc0000fe000, 0x12bed40, 0xc000134040, 0x0, 0xc000138030, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/request.go:645 +0x6e8
net/http.(*persistConn).writeLoop(0xc000110000)
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:1888 +0x1b8
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.11.2/libexec/src/net/http/transport.go:1339 +0x966

rax    0x104
rbx    0x2
rcx    0x7ffeefbff5e8
rdx    0x800
rdi    0x145a1a0
rsi    0x80100000900
rbp    0x7ffeefbff670
rsp    0x7ffeefbff5e8
r8     0x0
r9     0xa0
r10    0x0
r11    0x202
r12    0x145a1a0
r13    0x16
r14    0x80100000900
r15    0xd6435c0
rip    0x7fff5fe401b2
rflags 0x203
cs     0x7
fs     0x0
gs     0x0
exit status 2

Much appreciate any response and sorry for the time wasted if that had already be solved. I did spend some amount of time doing prior research and PoC before posting this.

@bradfitz bradfitz added this to the Go1.13 milestone Dec 14, 2018

@bradfitz bradfitz changed the title net/http hangs when io.Pipe is used as request body -- client net/http: Transport hangs when io.Pipe is used as request body Dec 14, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.