Join GitHub today
net/http: first byte of outgoing request body is sometimes lost #17071
(Apologies for not having been able to reproduce this hermetically. I suspect maybe it has something to do with SSL, which I don't know how to set up locally. I'm pretty sure the bug does lie in the Go client though; please keep reading.)
Here is a program that makes a bunch of calls to the Google Cloud Storage API. The calls are all identical. Although this is a heavily-reduced test case and they are not valid according to the GCS API (in particular they're missing credentials), they all do supply valid JSON in the request body.
Despite this, every time I run the program, after at most a few tens of seconds I receive an HTTP 400 parse error from GCS:
I happen to have insider knowledge that this charmingly terse error message from GCS indicates that the JSON in the request body could not be parsed. (If you want details, please contact me internally within Google.) In particular, the server on the other end does not receive the opening
Go's own HTTP2 debugging agrees.
With this invocation, among lots of other output I see
The http2 package has forgotten to write the opening curly brace. In contrast, everything works fine if I use
FYI, I have observed this from Sydney (Google office and at home, multiple computers) and from a computer within a Google data center in the US. I've seen this in Go 1.7.1 and Go 1.6.1, as well as at HEAD.
changed the title
net/http: first character of outgoing request body is sometimes lost
Sep 12, 2016
This probably warrants a Go 1.7.2 since it's sporadic data corruption. It happens whenever a Request.ContentLength == 0 && Request.Body != nil && the connection is either closed or temporarily at max-streams capacity.
Two possible workarounds, though:
Thanks for the quick fix. Please also consider a patch release for Go 1.6. Note that this bug was actually introduced in a patch release, since Go 1.6 didn't have HTTP2 on by default but 406752b made it into Go 1.6.2. (I may be confused about the details.)
The first Go 1.6 included HTTP/2 by default for all user-created http.Transports but not for the http.DefaultTransport. Go 1.6.2 fixed that. All the Go 1.6 betas before Go 1.6 had it enabled for the DefaultTransport too.
In any case, we don't issue point releases for two releases back. Only Go 1.7.x at this point.
referenced this issue
Sep 13, 2016
I don't have much more information. Just that it locked up in the client in a concurrent scenario and appeared to be sitting in those two places from the goroutine dumps. Tested with and without the commit but it was not a fully isolated test however seems highly likely since the commit moved that call past a lock.