runtime/pprof: inline frames may not use combined location

[prattmic]

What version of Go are you using (go version)?

$ go version
go version devel +a50c3ffbd4 linux/amd64

Does this issue reproduce with the latest release?

Regression in Go 1.14

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/usr/local/google/home/mpratt/.cache/go-build"
GOENV="/usr/local/google/home/mpratt/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/usr/local/google/home/mpratt/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/google-golang"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/google-golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build076660292=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Collect a heap profile from a profile with stacks truncated in the middle of inline frames: https://play.golang.org/p/6--6a10WPCa

What did you expect to see?

Consistent profile output.

What did you see instead?

http://golang.org/cl/204636 attempts to group inlined frames into the same Location, as required by the pprof format. Once a location is created, it is cached for reuse by future identical frames.

However, the runtime may truncate stacks, potentially truncating in between inlined frames. If the truncated frame is encountered first, then separate locations will be created for the two frames. If a complete frame is encountered first, then a combined location will be use for both frames.

This behavior is nondeterministic, so similar profiles may different:

This may be particularly problematic for tools looking at diffs between two profiles.

The former has location:

    30: 0x4bb6df M=1 main.depth1 /usr/local/google/home/mpratt/Downloads/alloc.go:18 s=0
             main.main /usr/local/google/home/mpratt/Downloads/alloc.go:312 s=0

The latter has locations:

    31: 0x4bb6df M=1 main.depth1 /usr/local/google/home/mpratt/Downloads/alloc.go:18 s=0
    33: 0x4bb6da M=1 main.main /usr/local/google/home/mpratt/Downloads/alloc.go:312 s=0

cc @hyangah @heschik

[hyangah]

@gopherbot please consider this for backport to 1.14, it's a regression.

[gopherbot]

Backport issue(s) opened: #37447 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[prattmic]

To clarify, there are two related issues here:

1. Go 1.14 will nondeterministically generate profiles with either combined or split stacks (the two screenshots above). 1.13 always generated split stacks in this case.
2. When the stacks are split, 1.14 loses the inline information on depth1. (1.13 never had inline information).

[gopherbot]

Change https://golang.org/cl/220377 mentions this issue: runtime/pprof: deterministic handling of truncated stacks

[gopherbot]

Change https://golang.org/cl/221577 mentions this issue: runtime/pprof: expand final stack frame to avoid truncation