Description
What version of Go are you using (go version
)?
$ go version go version go1.13.6 darwin/amd64
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/Users/jayden/Library/Caches/go-build" GOENV="/Users/jayden/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/jayden/Workspace/GoProject" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/usr/local/go" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/jayden/Workspace/GoProject/src/golang.org/x/net/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j1/94761d9d49j6gz5p55375sbr0000gn/T/go-build875938892=/tmp/go-build -gno-record-gcc-switches -fno-common"
What did you do?
I ran an http2 webdav server and I made a client with golang x/net/http2 for a test.
The client uploaded a 1 MB file infinitely with a few hundred goroutine.
MaxConcurrentStreams of the server was 100 and StrictMaxConcurrentStreams of the client was false.
I measured the throughput between the server and the client on AWS.
And network bandwidth between the server and the client was 150 Gbps in the infrastructure level.
What did you expect to see?
I expected that the throughput would be around 150 Gbps.
What did you see instead?
The measured throughput was 90 Gbps.
Proposal
I figured out what caused this. I modified the http2 connection pool a little as shown in the below link.
(ochanism/net@b2cc93d)
With the above code, I could achieve 150 Gbps throughput.
For achieving high throughput, multiple TCP connections are needed.
But the current version of the http2 connection pool increases TCP connections only when there is no available TCP connection (CurrentMaxStreams == MaxConcurrentStreams for all the connections).
I reduced MaxConcurrentStreams value at the server-side, but the number of TCP connections wasn't increased than I expected.
So I added a MinConcurrentConns field to http2 Transport.
MinConcurrentConns is the minimum number of TCP connections and ClientConnPool tries to maintain MinConcurrentConns TCP connections at least. If 0, ClientConnPool creates a TCP connection only when needed.
Please review my proposal.