net/http: http2 Transport retains Request.Body after request is complete, not GCed

[anacrolix]

I believe there's some kind of memory leak when using the following Client. b []byte below is typically 16KB at a time, and is never GC'd after having entered this section of code. Memory use climbs proportional to the number of requests sent on the client. I've ruled out the calling code. It didn't occur before I switched to h2 from HTTP/1.1. I assume some kind of h2 session-related item isn't being cleaned-up due to my unusual configuration.

Client: &http.Client{
                Transport: &http2.Transport{
                    TLSClientConfig: &tls.Config{
                        InsecureSkipVerify: true,
                        NextProtos:         []string{"h2"},
                    },
                },
            },

...

    req, err := http.NewRequest("PATCH", me.url, bytes.NewReader(b))
    if err != nil {
        return
    }
    req.Header.Set("Content-Range", fmt.Sprintf("bytes=%d-", me.off))
    req.ContentLength = int64(len(b))
    resp, err := me.fs.Client.Do(req)
    if err != nil {
        return
    }
    resp.Body.Close()

[bradfitz]

Interesting.

I wrote a test that reproduces this (in https://go-review.googlesource.com/18873) but I can't figure out what's retaining the memory. As far as I can tell, it should be GC'ed.

/cc @aclements @randall77 for help if either of you have time.

[bradfitz]

Interesting.

I wrote a test that reproduces this (in https://go-review.googlesource.com/18873) but I can't figure out what's retaining the memory. As far as I can tell, it should be GC'ed.

/cc @aclements @randall77 for help if either of you have time.

[anacrolix]

It also occurs on 1.5. I take it it's nontrvial to see what is referencing the bytes ?

[anacrolix]

It also occurs on 1.5. I take it it's nontrvial to see what is referencing the bytes ?

[bradfitz]

We have a heap dumper and tools to kinda view said heap but I'm not sure how to really use it these days.

[bradfitz]

We have a heap dumper and tools to kinda view said heap but I'm not sure how to really use it these days.

[gopherbot]

CL https://golang.org/cl/18873 mentions this issue.

[gopherbot]

CL https://golang.org/cl/18873 mentions this issue.

[randall77]

It's being held alive because it is in the reqCanceler map. Does that help any?
(in the list below, pointers go from bottom to top).

tracealloc(0xc820114ee0, 0x20, strings.Reader)
tracealloc(0xc820118270, 0x10, ioutil.nopCloser)
tracealloc(0xc8200f1260, 0xe0, http.Request)
tracealloc(0xc8200a42d0, 0x90, map.bucket[_http.Request]func()) (setReqCanceler: t.reqCanceler[r] = fn)
tracealloc(0xc820116900, 0x30, map.hdr[_http.Request]func()) (setReqCanceler: t.reqCanceler = make(map[*Request]func()))
tracealloc(0xc8200e80c0, 0xc0, http.Transport)
tracealloc(0xc82010c480, 0x60) // cst.c

[randall77]

It's being held alive because it is in the reqCanceler map. Does that help any?
(in the list below, pointers go from bottom to top).

tracealloc(0xc820114ee0, 0x20, strings.Reader)
tracealloc(0xc820118270, 0x10, ioutil.nopCloser)
tracealloc(0xc8200f1260, 0xe0, http.Request)
tracealloc(0xc8200a42d0, 0x90, map.bucket[_http.Request]func()) (setReqCanceler: t.reqCanceler[r] = fn)
tracealloc(0xc820116900, 0x30, map.hdr[_http.Request]func()) (setReqCanceler: t.reqCanceler = make(map[*Request]func()))
tracealloc(0xc8200e80c0, 0xc0, http.Transport)
tracealloc(0xc82010c480, 0x60) // cst.c

[bradfitz]

Perfect! That helps a ton.

How did you get that output?

[bradfitz]

Perfect! That helps a ton.

How did you get that output?

[randall77]

I added a print statement to print the address of the strings.Reader.
I ran the test with GODEBUG=allocfreetrace=1 and saved the output.
Then I put a heap dumper in and ran the view on the output.
Then I found the strings.Reader in question using the address printed above.
Then I walked back the object graph using the viewer, using the allocfreetrace dump to find the allocation site (and type) for each object .

[randall77]

I added a print statement to print the address of the strings.Reader.
I ran the test with GODEBUG=allocfreetrace=1 and saved the output.
Then I put a heap dumper in and ran the view on the output.
Then I found the strings.Reader in question using the address printed above.
Then I walked back the object graph using the viewer, using the allocfreetrace dump to find the allocation site (and type) for each object .

[anacrolix]

Great, thanks

[anacrolix]

Great, thanks

[anacrolix]

This doesn't seem to have fixed it.

[anacrolix]

This doesn't seem to have fixed it.

[anacrolix]

I had a play with your test. If I make the server response have an empty body, by removing the line
io.WriteString(w, "Hello.")
The h2 test will hang.

[anacrolix]

I had a play with your test. If I make the server response have an empty body, by removing the line
io.WriteString(w, "Hello.")
The h2 test will hang.

[bradfitz]

@anacrolix, nice! Confirmed. And that one's a bug in http2, not in the net/http integration. Easy fix: https://go-review.googlesource.com/19134

[bradfitz]

@anacrolix, nice! Confirmed. And that one's a bug in http2, not in the net/http integration. Easy fix: https://go-review.googlesource.com/19134

[gopherbot]

CL https://golang.org/cl/19134 mentions this issue.

[gopherbot]

CL https://golang.org/cl/19134 mentions this issue.

[gopherbot]

CL https://golang.org/cl/19135 mentions this issue.

[gopherbot]

CL https://golang.org/cl/19135 mentions this issue.