Skip to content

net/http: http.Get unnecessary data transfer #36242

Closed as not planned
Closed as not planned
@johnfitzz

Description

@johnfitzz

http roundtrippers transfer unnecessary amount of data even if nothing is read from the response body.

In my production code (which is not demonstrated here) I'm reading the first 10-15 bytes from a large file over http. However, http roundtrippers transfer unnecessary data. In the real code I'm using io.LimitedReader and/or CopyN or something to limit the data. However, this issue is not about how to transfer data efficiently, this issue is about roundtrippers' inefficient behavior.

What version of Go are you using (go version)?

go version go1.13 darwin/amd64

What version of OS X are you using?

ProductName:	Mac OS X
ProductVersion:	10.15.1
BuildVersion:	19B88

What did you do?

See the code on playground.
Note: I created this code only to demonstrate the issue.

resp, err := http.Get("https://raw.githubusercontent.com/IBM/MAX-Image-Resolution-Enhancer/master/samples/test_examples/original/airplane.png")
if err != nil {
	log.Fatal(err)
}
defer resp.Body.Close()

// Here, before the transfer, http2 roundtripper buffers up 150KB to 2MB or so of the airplane.png.
// Please see the code for details. 
// ...

What did you expect to see?

10 KiB is arbitrary here but I wanted to see less amount of data is being transferred.

$ nettop -m tcp -p $PID
                                                  bytes_in   bytes_out
gettest.2241                                      10 KiB     610 B
   tcp4 192.168.1.36:50082<->151.101.12.133:443   10 KiB     610 B

What did you see instead?

This is what I saw — it transferred unnecessary amount of data:

# when the transport is pconn.RoundTrip(req)
$ nettop -m tcp -p $PID
                                                 bytes_in   bytes_out
gettest.2241                                     178 KiB    610 B
   tcp4 192.168.1.36:50082<->151.101.12.133:443  178 KiB    610 B

# when the transport is pconn.alt.RoundTrip(req)
$ nettop -m tcp -p $PID
                                                 bytes_in   bytes_out
gettest.2241                                     2200 KiB   610 B
   tcp4 192.168.1.36:50082<->151.101.12.133:443  2200 KiB   610 B

Investigation:

transport.go#L530-L536 here, when it selects the alternative transport (pconn.alt.RoundTrip(req)), it buffers up a lot of data even before I read from the response.Body.

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions