-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: frame pointer unwinding can fail on system goroutines #63630
Comments
I looked into this a bit. It wouldn't be straightforward because, for example, the Other things I can think of:
|
There should be only a small number of TOPFRAME functions. Fixing them up manually sounds reasonable. If there are ones that are tricky or not possible to fix (like
I assume we need to keep a table of stack addresses (not PCs)? Maybe doable but it is probably hard to maintain that table, and as you mentioned, also hurt performance. Maybe we can check for whether the new FP falls out of the stack boundary and stop if so? But we also want to handle stack transitions in FP unwinding? Then maybe we'd need to check both |
Yeah, that might be more trouble than it's worth. IMO, if there's only Go code in the call chain, frame pointer unwinding should ideally work with no extra guardrails. Since we fully control the compiler and runtime, we can properly fix cases where it doesn't work.
Agreed, seems reasonable to me 👍 |
Change https://go.dev/cl/540476 mentions this issue: |
I have some experience now trying to do this for heap profiling. In the case of I saw this problem trying to do FP unwinding for heap profiling. For example, we might allocate a new M during I think we should probably have |
I agree in general it may not always be safe or sensible to unwind through stack transition. I think it may be reasonable to unwind through a user-to-system stack transition (like calling I don't know the specifics about Is there a case we do want to traceback through an |
Sorry, I left out some detail. The general pattern is that a goroutine calls
No, I don't think so. We have some execution tracing events recorded during |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Not exactly since we don't use frame pointer unwinding in this situation in any release.
However I believe the underlying issue is there in the latest release
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Attempted to use frame pointer unwinding to collect call stacks for the memory profiler. The motivation was in part to bring the efficiency gains we got for execution tracing to other places, but also to make frame pointers more widely used and thus more well-tested and reliable for the situations where we really want it to work. I observed this with patchset 1 of this CL and could reproduce it locally with
env GODEBUG=memprofilerate=1 go test ./runtime
.What did you expect to see?
Frame pointer unwinding to not fail.
What did you see instead?
Frame pointer unwinding fails (crashing) when collecting memory profile samples:
The unwinding fails after visiting the
mstart
frame. This is the entry point of the M. This function has aTOPFRAME
annotation, which helps unwinders using DWARF know not to proceed, but doesn't help for frame pointer unwinding. Should we could modify themstart
implementations to explicitly set the frame pointer register to 0 on architectures with frame pointers? That resolves this failure, at least. We could also teach the assembler to do that when it seesTOPFRAME
annotations.The text was updated successfully, but these errors were encountered: