cmd/pprof: incorrect CPU profile graph

[mokiat]

Go version

go version go1.25.3 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/momchil/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/momchil/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build906052103=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/momchil/Workspace/GoProjects/PROJECT/go.mod'
GOMODCACHE='/home/momchil/.go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/momchil/.go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/momchil/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK='/home/momchil/Workspace/GoProjects/go.work'
PKG_CONFIG='pkg-config'

What did you do?

I am developing a game project of mine written in Go and am using pprof to investigate CPU bottlenecks.

I am using the HTTP approach, so that I can hook up at any point in time.

import _ "net/http/pprof"

go func() {
	log.Println(http.ListenAndServe("localhost:6060", nil))
}()

(as shown in https://pkg.go.dev/net/http/pprof)

I use the following commands to fetch and analyize the profile.

curl -o cpu.pprof 'http://localhost:6060/debug/pprof/profile?seconds=30'
go tool pprof cpu.pprof

I use a Taskfile to start the project (should not be relevant):

  run:
    desc: Run the game.
    env:
      GODEBUG: cgocheck=0
    cmds:
      - go run './cmd/game' {{.CLI_ARGS}}

(The problem occurs even when GODEBUG is not set, so this flag is irrelevant)

What did you see happen?

When I use web to analyzie the profile graph I am seeing strange and incorrect relations.

For example, the sortLights method never calls the renderDirectionalLightShadowMaps method.
This part of the code is open-source and can be seen here:
https://github.com/mokiat/lacking/blob/bb73cd3d3ba73fd7be64c94c6790503c875e578e/game/graphics/stage_shadow.go#L52-L67

I am even seeing some methods appear in the graph that are top-level ones and are never used in the project .

What did you expect to see?

A valid performance chart.

Unfortunately, the main project I am running is closed-source and I can't provide it for testing.

Has anyone else stubled upon this? Is there something I might be doing wrong.
What's interesting is that it is not corrupt each time. Some pprof fetches are actually valid and make sense.

[gabyhelp]

Related Issues

• pprof: strange pprof heap flamegraph #56782 (closed)
• cmd/pprof: Profile is incorrect #16926 (closed)
• pprof - web, displays the recursive function on the picture incorrectly. #48644 (closed)
• runtime/pprof: shows invalid instruction addresses #70609 (closed)
• gccgo problems with pprof CPU profile #26595 (closed)
• pprof: Focus expression matched no samples #63572 (closed)
• proposal: runtime/pprof: add PMU-based profiles #36821
• runtime/pprof: use full file paths in source #45283
• pprof: go tool pprof cannot read the pprof file normally #66227 (closed)

Related Discussions

• 'go tool pprof' only shows "Type: CPU" and not a callgraph

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mokiat]

Not sure that the above issues match exactly (maybe one that was discarded for insufficient data).
I will try to reproduce this in the following days with an older open-source project of mine, by bumping it to Go 1.25 and hoping that it is enough to cause the problem.

[nsrip-dd]

Do you get different, correct-looking results if you run go tool pprof -symbolize=none cpu.pprof on an incorrect profile?

As far as I can tell, the pprof tool shouldn't try to symbolize if the profile already has file/line info. But if it did try to symbolize and the binary has changed since the profile was collected, that would definitely result in a nonsense profile.

[mokiat]

@nsrip-dd , yes, using -symbolize=none appears to fix the issue. At least the graph now makes sense, can't be 100% sure about the values.

As for the binary, I am using go run so it should end up somewhere in a temp directory. However, I keep the app running, so I expect that the binary is not changed/removed while using go tool pprof.

[dmitshur]

CC @golang/runtime, @cherrymui.