x/net/http2: window updates on randomWriteScheduler after stream closed cause memory leaks

[jared2501]

What version of Go are you using (go version)?

go version go1.12.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What did you do? What did you expect to see? What did you see instead?

I've been tracing down some issues in a creeping response times in a gRPC server that uses HTTP/2.

The big jump downwards is when I restart the server (purple=median, blue=p95, yellow=max response times).

After much poking and logging, I've traced it down to the entries in the randomWriteScheduler.sq growing, and not being garbage collected. What I believe is happening is that RST_STREAM frames are being sent on closed streams, which causes the randomWriteScheduler to allocate new entries in the randomWriteScheduler.sq map (see code here). The RST_STREAM frames will get queued and emitted by WriteScheduler.Pop() eventually, however since the stream has already been closed, they will never get deleted from the randomWriteScheduler.sq map since randomWriteScheduler.CloseStream will never be called, which causes this map to grow unbounded.

The priorityWriteScheduler handles this case by not allocating new nodes, but instead attaching RST_STREAM frames for closed streams to it's root node (in fact, it has a nice comment explaining this logic here).

I can look at providing a minimum repro of the issue if required, but it's a bit tricky to isolate since it requires delay in the stream to reproduce. Hopefully, the difference in behaviour between the randomWriteScheduler and the priorityWriteScheduler, and reading the code, is enough to identify & fix the issue here.

[jared2501]

I've run the gRPC server through a full traffic cycle with the priorityWriteScheduler enabled and it seems to have fixed the issue!

Here's a suggested fix to the leak:

diff --git a/http2/writesched_random.go b/http2/writesched_random.go
index 36d7919..4c34683 100644
--- a/http2/writesched_random.go
+++ b/http2/writesched_random.go
@@ -51,8 +51,14 @@ func (ws *randomWriteScheduler) Push(wr FrameWriteRequest) {
        }
        q, ok := ws.sq[id]
        if !ok {
-               q = ws.queuePool.get()
-               ws.sq[id] = q
+               // id is an idle or closed stream. wr should not be a HEADERS or
+               // DATA frame. However, wr can be a RST_STREAM. In this case, we
+               // push wr onto the zero write queue, rather than creating a new
+               // write queue, since RST_STREAM is tiny.
+               if wr.DataSize() > 0 {
+                       panic("add DATA on non-open stream")
+               }
+               q = &ws.zero
        }
        q.push(wr)
 }

[jared2501]

Quick ping @bradfitz @odeke-em - any chance of landing this in 1.14?

[jared2501]

Sorry to harp on this issue, but quick ping on this @bradfitz (now that you're back from leave - congrats btw!)

[bradfitz]

@jared2501, thanks for the great bug report & diagnosis!

@gopherbot, please backport to Go 1.13.

/cc @tombergan

[gopherbot]

Backport issue(s) opened: #34636 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/198462 mentions this issue: http2: fix memory leak in random write scheduler

[bradfitz]

@jared2501, can you try out https://golang.org/cl/198462 and report back about whether it helps or not? And any performance numbers if that's easy?

Also, code reviews welcome from anybody.

[gopherbot]

Change https://golang.org/cl/198617 mentions this issue: [release-branch.go1.13] http2: fix memory leak in random write scheduler

[gopherbot]

Change https://golang.org/cl/198897 mentions this issue: [release-branch.go1.13] net/http: update bundled http2 for memory leak fix

[jared2501]

@bradfitz change looks like a reasonable fix to me. Thanks! Will try and find a chance to run it some time next week.

I'll see what I can do re perf numbers, but I suspect we don't have good enough instrumentation to capture any difference here. Will include them if there is :)

[gopherbot]

Change https://golang.org/cl/200102 mentions this issue: net/http: update bundled x/net/http2