internal/poll: Splice() creates and destroys pipes in high frequency, wasting system resources #42740
Description
What version of Go are you using (go version)?
$ go version go version go1.15 linux/amd64
Does this issue reproduce with the latest release?
YES
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GO111MODULE="on" GOARCH="amd64" GOBIN="/root/dev/go/bin" GOCACHE="/root/.cache/go-build" GOENV="/root/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/root/dev/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/root/dev/go" GOPRIVATE="" GOPROXY="direct" GOROOT="/usr/lib/golang" GOSUMDB="off" GOTMPDIR="" GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/data/dev/golang/go/src/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build736136467=/tmp/go-build -gno-record-gcc-switches"
What did you do?
We were transmitting large amounts of data over the network, based on a mass of TCP sockets among goroutines.
The data transmission between TCP sockets in Go is optimized by the underlying splice() call on Linux which creates and destroys a new pipe every time at present, costing a mass of system resources and may slow down the performance.
For instance, if we started 50,000 concurrent socket goroutines to transfer data over the network, resulting in 64KB(pipe buffer on Linux) * 50,000 ≈ 3GB, it cost too many system resources and it may have the Processes OOM in extreme circumstances.
What did you expect to see?
Reusing the pipe buffers when calling splice(), save system resources, and improve performance.
What did you see instead?
High system resource consumption and inefficient processes.