Repository navigation
runtime/pprof: theoretical appendLocsForStack panic with SIGPROF between cgocallback and exitsyscall #70529
Description
runtime/pprof.appendLocsForStack asserts that an inline-expanded PC always expands to the same number of logical PCs (inlined frames). This is a static property of a given PC, so it should always be true.
When handling SIGPROF, if we are on an extra M running in C, we don't try to traceback since this is C anyway, and just add a sample with stack {pc, runtime._ExternalCode}.
"We are on an extra M running in C" is defined as gp.m.isExtraInC. In cgocallbackg, we clear this field after exitsyscall returns. This leaves a fairly long window when we are in fact running Go code, but the SIGPROF handler will think it is in C.
A lot of this code (particularly in exitsyscall) is reachable from normal Go code as well. If any of this code has more than 2 inlined frames at a single PC, then a SIGPROF from a normal Go context followed by a SIGPROF in this cgocallback context could trigger this appendLocsForStack panic.
I do not know if any code reachable in this window actually has more than 2 inlined frames. Only 2 frames is insufficient, as appendLocsForStack wouldn't actually care that the second frame is runtime._ExternalCode instead of the proper frame.
One potential fix is to attempt to do inline expansion in sigprofNonGoPC in case it actually is a Go PC.
Activity
gabyhelp commentedon Nov 22, 2024
Related Issues
Related Code Changes
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)
nsrip-dd commentedon Feb 19, 2025
I believe we're seeing this problem in one of our programs. We're using the Apache Arrow Go client library, which has C (and also maybe C++?) code that calls back into Go. I think it also calls Go from C on "extra" Ms, based on seeing samples in a CPU profile where cgocallback is the root frame. We've seen panics like this, on Go 1.23.6:
I unfortunately don't have the exact PC it's failing to handle. But I do see in CPU profiles for the program that this call to
casgstatusinexitsyscallis inlined, and there's aCompareAndSwapinlined into that call:So I think we meet the conditions described in this issue.
prattmic commentedon Feb 21, 2025
I filed this issue when investigating one of these
appendLocsForStackpanics. I came up with this theoretical problem, though it ended up not being the cause of my crash.For that crash, I modified
sigprofNonGoPCto throw if the PC actually is a Go PC.Something like
If you can reproduce, something like this might be useful as you can catch the problem at the moment it occurs rather than much later when the panic occurs.
dudusmida commentedon Aug 22, 2025
go1.25 still panic, but only with panic information is
it seems that it is still the same problem.
go1.25-commit: 89228ca
prattmic commentedon Sep 10, 2025
@dudusmida If you still have the binary, could you disassemble it (
go tool objdump) and copy the function(s) containing the addresses instk = [0x4bf3c0 0x4c0ac1], l.pcs = [0x4bf3c0 0x4bf3c1 0x4bf3c7]?harshavardhana commentedon Nov 12, 2025
@prattmic got bit by this recently with MinIO compiled with some RDMA/Shared library dependencies while performing a CPU profile
The addresses
0x45bf20,0x45bf21, and0x45bf27are all withinruntime.exitsyscallfast:Hopefully this is of some use.
prattmic commentedon Nov 12, 2025
Thanks, that does seem useful. For completeness, where is
0x45d681? Is that inruntime.exitsyscall?runtime/pprof: avoid panic in appendLocsForStack for sigprofNonGoPC s…
runtime/pprof: avoid panic for sigprofNonGoPC short stacks
runtime/pprof: avoid panic for sigprofNonGoPC short stacks
gopherbot commentedon Aug 15, 2026
Change https://go.dev/cl/815820 mentions this issue:
runtime/pprof: avoid panic for sigprofNonGoPC short stacksruntime/pprof: avoid panic for sigprofNonGoPC short stacks