Description
http roundtrippers transfer unnecessary amount of data even if nothing is read from the response body.
In my production code (which is not demonstrated here) I'm reading the first 10-15 bytes from a large file over http. However, http roundtrippers transfer unnecessary data. In the real code I'm using io.LimitedReader and/or CopyN or something to limit the data. However, this issue is not about how to transfer data efficiently, this issue is about roundtrippers' inefficient behavior.
What version of Go are you using (go version
)?
go version go1.13 darwin/amd64
What version of OS X are you using?
ProductName: Mac OS X ProductVersion: 10.15.1 BuildVersion: 19B88
What did you do?
See the code on playground.
Note: I created this code only to demonstrate the issue.
resp, err := http.Get("https://raw.githubusercontent.com/IBM/MAX-Image-Resolution-Enhancer/master/samples/test_examples/original/airplane.png")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
// Here, before the transfer, http2 roundtripper buffers up 150KB to 2MB or so of the airplane.png.
// Please see the code for details.
// ...
What did you expect to see?
10 KiB is arbitrary here but I wanted to see less amount of data is being transferred.
$ nettop -m tcp -p $PID
bytes_in bytes_out
gettest.2241 10 KiB 610 B
tcp4 192.168.1.36:50082<->151.101.12.133:443 10 KiB 610 B
What did you see instead?
This is what I saw — it transferred unnecessary amount of data:
# when the transport is pconn.RoundTrip(req)
$ nettop -m tcp -p $PID
bytes_in bytes_out
gettest.2241 178 KiB 610 B
tcp4 192.168.1.36:50082<->151.101.12.133:443 178 KiB 610 B
# when the transport is pconn.alt.RoundTrip(req)
$ nettop -m tcp -p $PID
bytes_in bytes_out
gettest.2241 2200 KiB 610 B
tcp4 192.168.1.36:50082<->151.101.12.133:443 2200 KiB 610 B
Investigation:
transport.go#L530-L536 here, when it selects the alternative transport (pconn.alt.RoundTrip(req)
), it buffers up a lot of data even before I read from the response.Body.