x/net/http2: decrease memory consumption when small request coming

[ayanamist]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build030072102=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

package main

import (
	"fmt"
	"io"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"time"

	"golang.org/x/net/http2"
)

func main() {
	ln, err := net.Listen("tcp", ":8088")
	if err != nil {
		log.Fatal(err)
	}

	http.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		n, _ := io.Copy(ioutil.Discard, r.Body)
		w.Write([]byte(fmt.Sprintf("%d from %v", n, r.Proto)))
	}))

	httpServer := &http.Server{
		ReadTimeout:  60 * time.Second,
		WriteTimeout: 60 * time.Second,
	}
	h2ServeOpts := &http2.ServeConnOpts{
		BaseConfig: httpServer,
	}
	h2Server := http2.Server{
		MaxReadFrameSize: 16 * 1024,
		IdleTimeout:      60 * time.Second,
	}

	for {
		conn, err := ln.Accept()
		if err != nil {
			log.Printf("accept: %v", err)
			continue
		}
		go h2Server.ServeConn(conn, h2ServeOpts)
	}
}

Use following to benchmark:
h2load -n 10000000 -c 1000 -m 100 -d ~/testdata http://127.0.0.1:8088/
~/testdata file is 950 bytes large.

What did you expect to see?

Use ps and top to see memory consumption, and not very large.

What did you see instead?

Memory consumption can go up to 8GB

[ayanamist]

After some pprof investigate, i found the problem is related to reqBuf used in stream, i have some optimization for such small request (request body less than 1KB).

[gopherbot]

CL https://golang.org/cl/34754 mentions this issue.

[ayanamist]

In our benchmark, after modification, heap size is no large than 1GB.

[rsc]

/cc @bradfitz

[tombergan]

This is effectively a dup of #16512, especially this comment, which notes that we'll need a smarter buffering strategy to support connection-wide flow control. With the suggested limit of 1MB per connection from #16512, and 1000 concurrent connections (h2load -c 1000 ...), that is a max of 1GB total, same as in the above comment.

[ayanamist]

@tombergan Although the result is similar with #16512, the implementation is quite different, i do not use any per connection memory limit, but use existing Content-Length header from requests. If i change the size of testdata to smaller like 100 bytes or larger like 4KB, the heap size will change too.
In fact there are many similar memory wasting, and this is the easiest modification i found.

[tombergan]

Yes, thanks for the report. We need to rethink the buffering strategy to better share the connection's flow window across all requests. As you point out, we should minimize the amount of memory allocated to individual requests, rather than blindly allocating 64KB to every request. However, I think we'd implement that slightly differently than in your sample CL, by trying to do a better job of reusing buffers for multiple requests so each POST doesn't require an allocation.

[ayanamist]

@tombergan I am very looking foward to your job, we have a very h2 traffic and quite need any cpu and memory optimization.

[gopherbot]

CL https://golang.org/cl/37400 mentions this issue.