Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: panic in appendLocsForStack on linux-arm64-packet #35538

Closed
bcmills opened this issue Nov 12, 2019 · 4 comments
Closed

runtime/pprof: panic in appendLocsForStack on linux-arm64-packet #35538

bcmills opened this issue Nov 12, 2019 · 4 comments
Assignees
Milestone

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 12, 2019

panic: runtime error: slice bounds out of range [6:5]

goroutine 245 [running]:
runtime/pprof.(*profileBuilder).appendLocsForStack(0x400014d080, 0x400022e140, 0x0, 0x8, 0x40001f6568, 0x5, 0x353, 0x0, 0x4, 0x8)
	/workdir/go/src/runtime/pprof/proto.go:397 +0x5b8
runtime/pprof.(*profileBuilder).build(0x400014d080)
	/workdir/go/src/runtime/pprof/proto.go:362 +0xfc
runtime/pprof.profileWriter(0x24bd60, 0x40004bc240)
	/workdir/go/src/runtime/pprof/pprof.go:779 +0xd8
created by runtime/pprof.StartCPUProfile
	/workdir/go/src/runtime/pprof/pprof.go:750 +0x128
FAIL	runtime/pprof	4.052s

2019-11-12T05:24:00-d22b573/linux-arm64-packet
2019-11-12T01:09:31-0ee22d9/linux-arm64-packet
2019-11-08T18:00:31-c444ec3/linux-arm64-packet

CC @hyangah @randall77

@bcmills bcmills added this to the Go1.14 milestone Nov 12, 2019
@hyangah

This comment has been minimized.

Copy link
Contributor

@hyangah hyangah commented Nov 12, 2019

Using gomote, printed out the the cached location pcs and the pcs in the stack. It's flaky - all failures I got are only from TestGoroutineSwitch so far. @randall77

=== RUN   TestGoroutineSwitch
XXX
Cached PCs: [300657 300665 300656 300665 300656 300665]
	[{300656  runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
	[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
	[{300655  runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
	[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
	[{300655  runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
	[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
vs New PCs: [300657 300665 300656 300665]
	[{300656  runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
	[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
	[{300655  runtime.nanotime /workdir/go/src/runtime/time_nofake.go 19 300416 {0x28dac8 0x386c80}}]
	[{300664 0x28dac8 runtime.checkTimers /workdir/go/src/runtime/proc.go 2623 300416 {0x28dac8 0x386c80}}]
cached loc = 19
panic: runtime error: slice bounds out of range [6:4]

goroutine 251 [running]:
runtime/pprof.(*profileBuilder).appendLocsForStack(0x40001eef20, 0x4000454d00, 0x0, 0x8, 0x4041192478, 0x4, 0x371, 0x0, 0x6, 0x8)
	/workdir/go/src/runtime/pprof/proto.go:410 +0x8ac
runtime/pprof.(*profileBuilder).build(0x40001eef20)
	/workdir/go/src/runtime/pprof/proto.go:362 +0xfc
runtime/pprof.profileWriter(0x24be00, 0x404118aea0)
	/workdir/go/src/runtime/pprof/pprof.go:779 +0xd8
created by runtime/pprof.StartCPUProfile
	/workdir/go/src/runtime/pprof/pprof.go:750 +0x128
FAIL	runtime/pprof	31.955s
@hyangah

This comment has been minimized.

Copy link
Contributor

@hyangah hyangah commented Nov 12, 2019

BTW, I just got an internal report on the same panic, but with linux-amd64, and with the heap profile.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 12, 2019

Change https://golang.org/cl/206958 mentions this issue: runtime/pprof: fix the inlined frame merge logic

gopherbot pushed a commit that referenced this issue Nov 14, 2019
tryAdd shouldn't succeed (and accept the new frame) if the last
existing frame on the deck is not an inlined frame.

For example, when we see the followig stack
[300656 300664 300655 300664]

with each PC corresponds to

[{PC:300656 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300655 Func:nil Function:runtime.nanotime File:/workdir/go/src/runtime/time_nofake.go Line:19 Entry:300416 {0x28dac8 0x386c80}}]
[{PC:300664 Func:0x28dac8 Function:runtime.checkTimers File:/workdir/go/src/runtime/proc.go Line:2623 Entry:300416 {0x28dac8 0x386c80}}]

PC:300656 and PC:300664 belong to a single location entry,
but the bug in the current tryAdd logic placed the entire stack into one
location entry.

Also adds tests - this crash is a tricky case to test because I think it
should happen with normal go code. The new TestTryAdd simulates it by
using fake call sequences. The test crashed without the fix.

Update #35538

Change-Id: I6d3483f757abf4c429ab91616e4def90832fc04a
Reviewed-on: https://go-review.googlesource.com/c/go/+/206958
Reviewed-by: Keith Randall <khr@golang.org>
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 18, 2019

Change https://golang.org/cl/207609 mentions this issue: runtime/pprof: avoid crash due to truncated stack traces

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.