runtime: poor performance in simulated environment

[Hendrik-H]

What version of Go are you using (go version)?

$ go version
go version go1.19.2 linux/s390x

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
AR="ar"
CC="s390x-linux-gnu-gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1969591831=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I wrote a little test tool that start 50 goroutines that waste some time and exits once all goroutines are done.
main.go.txt

What did you expect to see?

When running the code on real hardware it finishes in about 200ms. I expected it to execute in roughly the same time in a virtual environment. We are using cecsim, which is based on KVM.

What did you see instead?

In our virtual environment the execution takes at least 10 times longer and the system shows a very high steal time. When using GODEBUG=asyncpreemptoff=1 the problem does not seem to ocur.
Here is an strace of a run on real hardware and one in a simulation:
testtool-1.19.2-strace-real-machine.txt
testtool-1.19.2-strace-sim.txt

The trace shows many nanosleep calls. Or assumption is that this might be the cause of the problem.

[prattmic]

One obvious difference is that the simulated version makes lots of clock_gettime system calls. Normally those are handled by the VDSO in userspace. Making full system calls will certainly slow things down, though 10x is much more extreme than I would expect.

cc @golang/s390x

[mknyszek]

CC @golang/runtime

[mknyszek]

I'm not sure what we can do here. We can try to call nanotime less but that's a pretty massive change to the runtime at this point. To clarify, what do you mean by "high steal time"? Also, given that the issue mainly only arises with GODEBUG=asyncpreemptoff=0, it may be that the simulated environment doesn't handle the OS signals the runtime sends very well. I'm not sure what we can do here about this on our side.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)