Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

internal/poll: Splice() creates and destroys pipes in high frequency, wasting system resources #42740

Open
panjf2000 opened this issue Nov 20, 2020 · 3 comments

Comments

@panjf2000
Copy link
Contributor

@panjf2000 panjf2000 commented Nov 20, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15 linux/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/root/dev/go/bin"
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/dev/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/dev/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/data/dev/golang/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build736136467=/tmp/go-build -gno-record-gcc-switches"

What did you do?

We were transmitting large amounts of data over the network, based on a mass of TCP sockets among goroutines.
The data transmission between TCP sockets in Go is optimized by the underlying splice() call on Linux which creates and destroys a new pipe every time at present, costing a mass of system resources and may slow down the performance.

For instance, if we started 50,000 concurrent socket goroutines to transfer data over the network, resulting in 64KB(pipe buffer on Linux) * 50,000 ≈ 3GB, it cost too many system resources and it may have the Processes OOM in extreme circumstances.

What did you expect to see?

Reusing the pipe buffers when calling splice(), save system resources, and improve performance.

What did you see instead?

High system resource consumption and inefficient processes.

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 20, 2020

Change https://golang.org/cl/271537 mentions this issue: internal/poll: implement a pipe pool for splice() call

@ALTree ALTree changed the title interanl/poll: Splice() creates and destroys pipes in high frequency, costing an amount of system resources internal/poll: Splice() creates and destroys pipes in high frequency, costing an amount of system resources Nov 20, 2020
@ALTree ALTree added this to the Go1.17 milestone Nov 20, 2020
@ALTree ALTree added the Performance label Nov 20, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 20, 2020

Reusing pipe descriptors makes sense to me, but I don't understand your example. Currently the pipes, and their buffers, only exist as long as they are being used. Even with a pipe pool, if you have 50,000 concurrent routines using splice, you are going to be using all that memory and will face the risk of an OOM. Having a pool doesn't reduce the resources required by concurrent use.

@panjf2000
Copy link
Contributor Author

@panjf2000 panjf2000 commented Nov 20, 2020

My initial thought was to create a pipe pool that limits the total number of pipes and blocks or fails if it runs out of the capacity of the pool, like HAProxy: https://github.com/haproxy/haproxy/blob/v2.4-dev0/src/pipe.c#L66-L67 which does save system resources, but after a second thought, I reckon that is not a rational choice for Go (as a programming language), so I think it would be a better idea for a pipe pool to just cache rather than limit pipes.

@panjf2000 panjf2000 changed the title internal/poll: Splice() creates and destroys pipes in high frequency, costing an amount of system resources internal/poll: Splice() creates and destroys pipes in high frequency, wasting system resources and slow down performance Nov 20, 2020
@panjf2000 panjf2000 changed the title internal/poll: Splice() creates and destroys pipes in high frequency, wasting system resources and slow down performance internal/poll: Splice() creates and destroys pipes in high frequency, wasting system resources Nov 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.