Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: poor performance in simulated environment #56704

Open
Hendrik-H opened this issue Nov 11, 2022 · 3 comments
Open

runtime: poor performance in simulated environment #56704

Hendrik-H opened this issue Nov 11, 2022 · 3 comments
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@Hendrik-H
Copy link

Hendrik-H commented Nov 11, 2022

What version of Go are you using (go version)?

$ go version
go version go1.19.2 linux/s390x

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
AR="ar"
CC="s390x-linux-gnu-gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1969591831=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I wrote a little test tool that start 50 goroutines that waste some time and exits once all goroutines are done.
main.go.txt

What did you expect to see?

When running the code on real hardware it finishes in about 200ms. I expected it to execute in roughly the same time in a virtual environment. We are using cecsim, which is based on KVM.

What did you see instead?

In our virtual environment the execution takes at least 10 times longer and the system shows a very high steal time. When using GODEBUG=asyncpreemptoff=1 the problem does not seem to ocur.
Here is an strace of a run on real hardware and one in a simulation:
testtool-1.19.2-strace-real-machine.txt
testtool-1.19.2-strace-sim.txt

The trace shows many nanosleep calls. Or assumption is that this might be the cause of the problem.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 11, 2022
@prattmic
Copy link
Member

prattmic commented Nov 11, 2022

One obvious difference is that the simulated version makes lots of clock_gettime system calls. Normally those are handled by the VDSO in userspace. Making full system calls will certainly slow things down, though 10x is much more extreme than I would expect.

cc @golang/s390x

@prattmic prattmic added the arch-s390x Issues solely affecting the s390x architecture. label Nov 11, 2022
@prattmic prattmic added this to the Backlog milestone Nov 11, 2022
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 11, 2022
@mknyszek
Copy link
Contributor

mknyszek commented Nov 11, 2022

CC @golang/runtime

@mknyszek
Copy link
Contributor

mknyszek commented Nov 16, 2022

I'm not sure what we can do here. We can try to call nanotime less but that's a pretty massive change to the runtime at this point. To clarify, what do you mean by "high steal time"? Also, given that the issue mainly only arises with GODEBUG=asyncpreemptoff=0, it may be that the simulated environment doesn't handle the OS signals the runtime sends very well. I'm not sure what we can do here about this on our side.

@mknyszek mknyszek added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-s390x Issues solely affecting the s390x architecture. compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

4 participants