runtime: CL 501976 performance and jitter regression

[JetSetIlly]

Go version

go version go1.22.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/steve/.cache/go-build'
GOENV='/home/steve/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/steve/Go/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/steve/Go/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/steve/Go/go1.22.0/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/home/steve/Go/go1.22.0/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build502182501=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I am comparing the performance my project when compiled with Go 1.22.0 and when compiled with Go 1.21.7

The full instructions for my testing procedure and the result CPU profiles and the test program have been placed in a specially created Github repository

What did you see happen?

The performance of my project has dropped significantly with v1.22.0. Using the metric of performance used in my project, the performance with Go v1.21.7 is 146.28fps and the performance with Go v1.22.0 is 133.48fps. This is a drop of 8.75%

What did you expect to see?

I expected either no significant change or an increased performance rating. This method of measurement has been used for many years and has shown increases in performance as expected when new Go versions have been released. For example, the introduction of profile-guided optimisation, showed the expected increase in performance.

[Jorropo]

This too https://www.reddit.com/r/golang/comments/1amnj7i/go_122_yielding_a_18_regression_in/

[randall77]

I need some more specific instructions to reproduce.
So far:

git clone https://github.com/JetSetIlly/Gopher2600
cd Gopher2600
git checkout d1482c139ab9c78b18902426e87367a58cf8b101
~/go1.22.0/bin/go build

That last build produces some strange results:

go: downloading github.com/pkg/term v1.1.0
go: downloading github.com/go-gl/gl v0.0.0-20211210172815-726fda9656d6
go: downloading github.com/inkyblackness/imgui-go/v4 v4.7.0
go: downloading golang.org/x/sys v0.8.0
go: downloading github.com/sahilm/fuzzy v0.1.0
go: downloading github.com/veandco/go-sdl2 v0.4.35
go: downloading golang.org/x/image v0.13.0
go: downloading github.com/go-audio/wav v1.1.0
go: downloading github.com/hajimehoshi/go-mp3 v0.3.4
go: downloading github.com/go-audio/audio v1.0.0
go: downloading github.com/go-audio/riff v1.0.0
# github.com/go-gl/gl/v3.2-core/gl
# [pkg-config --cflags  -- gl gl]
Package gl was not found in the pkg-config search path.
Perhaps you should add the directory containing `gl.pc'
to the PKG_CONFIG_PATH environment variable
Package 'gl', required by 'virtual:world', not found
Package 'gl', required by 'virtual:world', not found
# github.com/veandco/go-sdl2/sdl
# [pkg-config --cflags  -- sdl2]
Package sdl2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `sdl2.pc'
to the PKG_CONFIG_PATH environment variable
Package 'sdl2', required by 'virtual:world', not found
# github.com/inkyblackness/imgui-go/v4
In file included from wrapper.cpp:7:
../../../go/pkg/mod/github.com/inkyblackness/imgui-go/v4@v4.7.0/imgui/imgui_draw.cpp: In function ‘void ImFontAtlasBuildPackCustomRects(ImFontAtlas*, void*)’:
../../../go/pkg/mod/github.com/inkyblackness/imgui-go/v4@v4.7.0/imgui/imgui_draw.cpp:2623:11: warning: ‘void* memset(void*, int, size_t)’ specified size between 18446744071562067968 and 18446744073709551615 exceeds maximum object size 9223372036854775807 [-Wstringop-overflow=]
 2623 |     memset(pack_rects.Data, 0, (size_t)pack_rects.size_in_bytes());
      |     ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Is there some PKG_CONFIG_PATH thing I need to set?
Is there some specific C compiler I need?
I also don't see a Cosmcark.bin file anywhere, which I need to run it?

[JetSetIlly]

I need some more specific instructions to reproduce. So far:

git clone https://github.com/JetSetIlly/Gopher2600
cd Gopher2600
git checkout d1482c139ab9c78b18902426e87367a58cf8b101
~/go1.22.0/bin/go build

That last build produces some strange results:

The full project uses third-party C libraries. They aren't used when running the performance measurement but are needed for compilation. (The libraries are used for the GUI but we don't need a GUI when doing measurement)

I'll prepare a minimal project tree that only does the performance measurement so as to make it easier to compile and test.

Is there some PKG_CONFIG_PATH thing I need to set? Is there some specific C compiler I need? I also don't see a Cosmcark.bin file anywhere, which I need to run it?

The Cosmcark.bin file is in https://github.com/JetSetIlly/gopher2600_performance_profiles

[Jorropo]

@JetSetIlly since you already have everything setup can you benchmark GOEXPERIMENT=loopvar with go1.21.7 please ?

[JetSetIlly]

@randall77 If you clone this project now and run 'go build .' it should compile without requiring any external C libraries (ie. no SDL or OpenGL). There's also no need to worry about the Cosmcark.bin

https://github.com/JetSetIlly/gopher2600_performance_profiles

The measurements are slightly different to the previous results but the point is that the measurement for 1.22.0 is consistently and significantly lower than 1.21.7

@Jorropo I tried that and the results are the same/similar to 1.21.7.

[Jorropo]

https://github.com/JetSetIlly/gopher2600_performance_profile 404s, is it private ?

[JetSetIlly]

https://github.com/JetSetIlly/gopher2600_performance_profile 404s, is it private ?

No. I missed the s off the end when I was copying and pasting. Doh!
https://github.com/JetSetIlly/gopher2600_performance_profiles

[randall77]

I am not seeing any significant performance differences between 1.21.6 and 1.22.0 (and tip, for that matter).
There's quite a bit of noise in this benchmark, so it is hard to be sure.

1.21.6

238.56 fps (71567 frames in 300.00 seconds) 397.6%
228.48 fps (68544 frames in 300.00 seconds) 380.8%
236.91 fps (71072 frames in 300.00 seconds) 394.8%

1.22.0

232.57 fps (69771 frames in 300.00 seconds) 387.6%
224.25 fps (67274 frames in 300.00 seconds) 373.7%
234.01 fps (70203 frames in 300.00 seconds) 390.0%

model name : 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz

[JetSetIlly]

@randall77 Thanks for looking at this.

If we take an average of the runs for 1.21.6 and 1.22.0 then there's a drop of 1.86% in performance. I would say that's significant. At least, I've never seen drops in performance like this between go versions before. I would agree that it's hard to get consistent measurements, but there is a consistency in a drop between the two versions.

On my hardware the drop is a lot larger (Intel(R) Core(TM) i3-3225 CPU @ 3.30GHz) as previously stated.

The test runs for 5 minutes but it doesn't need to run that long. On line 22 of main.go, there's a string specifying the duration. It's currently "5m". You can always lower that if it makes it easier.

If there's anything I can do to reduce the benchmark noise let me know.

[randall77]

If we take an average of the runs for 1.21.6 and 1.22.0 then there's a drop of 1.86% in performance.

Sure, but the run-to-run variance can be as much as 10%. It's not clear to me that the 1.86% drop is real. We'd need a lot more runs to have any confidence that the 1.86% drop isn't just noise.

On my hardware the drop is a lot larger (Intel(R) Core(TM) i3-3225 CPU @ 3.30GHz) as previously stated.

This is a possible cause. That chip is quite a bit older than the one I'm using. (~12 years vs ~4 years old.)
One possibility, though I think unlikely, is that your chip is missing some instruction (POPCNT?) that we started using more in 1.22. What does /proc/cpuinfo list for your flags?
Maybe there is something that got a bit larger and no longer fits in the small cache of that chip? Again, probably unlikely.
It would help if you could try on a more recent chip to see if that illuminates anything.

If there's anything I can do to reduce the benchmark noise let me know.

It would help a lot with investigations if we could get the benchmark variance down. Can you make any progress on that front? The benchmark appears single-threaded and doesn't allocate much, so I'm kind of confused as to where the variance might be coming from. (It is even possible the source of variance and the source of slowdown are related somehow.)

[JetSetIlly]

If we take an average of the runs for 1.21.6 and 1.22.0 then there's a drop of 1.86% in performance.

Sure, but the run-to-run variance can be as much as 10%. It's not clear to me that the 1.86% drop is real. We'd need a lot more runs to have any confidence that the 1.86% drop isn't just noise. But I'm confident that there is.

I understand. My view of the code is very different to yours of course, so I'm perhaps more sensitive to the variation. I'll do some more runs here to assure myself that there is a real effect here.

On my hardware the drop is a lot larger (Intel(R) Core(TM) i3-3225 CPU @ 3.30GHz) as previously stated.

This is a possible cause. That chip is quite a bit older than the one I'm using. (~12 years vs ~4 years old.) One possibility, though I think unlikely, is that your chip is missing some instruction (POPCNT?) that we started using more in 1.22. What does /proc/cpuinfo list for your flags? Maybe there is something that got a bit larger and no longer fits in the small cache of that chip? Again, probably unlikely. It would help if you could try on a more recent chip to see if that illuminates anything.

This CPU does have POPCNT

It would help a lot with investigations if we could get the benchmark variance down. Can you make any progress on that front? The benchmark appears single-threaded and doesn't allocate much, so I'm kind of confused as to where the variance might be coming from. (It is even possible the source of variance and the source of slowdown are related somehow.)

This is interesting. If I run with GOMAXPROCS=1 then the variation between the versions goes away.

I'll do a lot more runs/tests here but with a small sample I can see very clearly that with v1.22.0 GOMAXPROCS=1 performs the same as 1.27.1 and GOMAXPROCS=n (ie. not set) the performance is lower by an average of 6%

With 1.27.1, the performance is the same regardless of the GOMAXPROCS value

[JetSetIlly]

I've updated the test program to output a rating every second, running for a total of one minute. This gives a clearer picture of what's happening I think.

An example run of the four variations, shows that 1.21.7 is consistent regardless of GOMAXPROCS but that with 1.22.0 there is a significant difference.

Table of results

Version	1.22.0	1.22.0	1.21.7	1.21.7
GOMAXPROCS	n	1	n	1
	145.88	146.46	148.85	146.91
	139.97	147.01	147.00	145.41
	147.05	145.42	148.03	146.32
	145.04	143.63	147.07	145.69
	140.05	144.52	146.92	145.16
	139.92	144.27	148.08	144.53
	140.06	144.93	147.94	145.33
	132.00	144.81	147.08	145.67
	137.95	144.80	147.99	144.99
	130.95	145.36	145.00	146.34
	139.11	144.51	145.91	145.73
	133.88	145.58	146.10	146.44
	137.07	145.16	145.87	146.41
	138.03	144.43	147.05	145.58
	129.89	144.84	146.05	146.00
	139.03	145.61	144.93	146.09
	137.08	144.71	146.07	144.74
	136.98	145.19	144.92	145.42
	137.90	143.95	146.05	146.17
	122.04	145.56	146.07	145.94
	130.03	144.45	144.91	143.91
	132.99	143.73	145.07	142.98
	143.96	143.31	144.94	145.41
	138.06	144.21	145.96	145.16
	137.95	144.24	147.05	144.81
	134.96	143.93	146.04	146.34
	143.07	145.27	147.01	146.04
	138.05	145.33	145.01	145.51
	133.92	143.37	144.94	145.98
	143.03	143.73	145.04	145.94
	144.08	145.12	146.01	145.25
	144.91	145.42	145.95	146.55
	130.02	144.66	145.98	143.76
	144.06	141.42	144.04	144.31
	132.89	144.09	145.91	145.56
	141.04	143.29	144.07	143.15
	140.00	145.02	147.05	145.38
	141.07	144.05	143.91	145.87
	140.90	145.04	145.04	145.07
	139.04	143.96	144.95	139.06
	140.06	145.04	146.07	147.02
	138.90	145.05	145.90	148.02
	147.06	143.95	145.08	147.06
	146.92	141.06	146.03	147.01
	144.07	145.04	146.02	147.06
	133.06	143.94	144.92	147.05
	138.00	144.39	146.05	148.02
	137.95	144.96	146.03	147.00
	138.90	145.06	145.94	147.04
	133.02	144.05	145.93	147.01
	138.11	144.98	145.00	147.99
	146.94	145.00	145.04	144.05
	140.05	145.06	144.07	147.05
	137.99	143.95	145.01	147.01
	144.00	143.08	145.91	147.03
	138.93	145.05	146.06	148.05
	146.05	144.05	146.00	148.02
	146.93	143.95	144.92	147.05
	145.06	144.04	145.12	147.05
	145.03	144.05	144.95	147.02
	137.91	143.96	145.94	147.06
	128.04	145.05	140.11	145.99
Average	138.98	144.50	145.77	145.91
Min/Max Difference	25.02	5.95	8.74	8.99