Repository navigation
proposal: runtime: expose column data on tracebacks #81629
Description
Proposal Details
Background
Runtime stack traces, like those produced by panics, or by functions such as runtime.Stack, produce a detailed output of frames with information like the filename, line number, and a program counter offset. An example:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: no panic at correct location
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x47d94b]
goroutine 1 gp=0x1e18437a41e0 m=0 mp=0x54b120 [running]:
panic({0x537ec0?, 0x486030?})
/home/urandom/projects/go/src/runtime/panic.go:878:12 +0x151 fp=0x1e184381ecf0 sp=0x1e184381ec48
pc=0x475211
Sometimes, this information is insufficient, due to the lack of column data within the output.
Proposal
I propose that the compiler have another PC table mapping the counter delta to the column of each instruction, similar to what is done for files and lines. The runtime will in turn read values from the table when constructing traceback data.
It is expected that the additional data will result in larger binary sizes. For storage-constrained systems, it should be possible to disable the collection of data via a new compiler debug flag.
Implementation
I've so far implemented this on top of the existing generic list of pcdata tables that exist per symbol in Pcln — mostly to keep the change small. Compiling some on-hand projects (from prometheus, to some smaller toy ones), both with and without column data, so far shows that the difference in size is about 2-5%
Activity
urandom commentedon Sep 21, 2026
The change, with which I tested the size differences:
https://go-review.googlesource.com/c/go/+/836065
gopherbot commentedon Sep 21, 2026
Change https://go.dev/cl/836065 mentions this issue:
runtime: add support for printing columns in tracebacksianlancetaylor commentedon Sep 21, 2026
The column information is only useful if there are multiple locations in the line that could appear in a stack trace. Except for the point of the initial panic, that can only be a function call. At the point of the initial panic, it could be a call to
panicindexand similar functions. It might be helpful to only record column information when it might be ambiguous.In general, since the column will only occasionally make a difference in interpreting a stack trace, personally I don't think it's worth 2 to 5% in binary size.
prattmic commentedon Sep 21, 2026
I'm not sure if column numbers are worth the cost, but some other use cases that have come up:
Column numbers in pprof profiles came up recently in #57175 (comment) as a a possible generalized improvement to help with #81392. i.e., with column numbers it might be more clear that a copy is expensive vs some other operation on the line.
For PGO builds, there has been on-and-off investigation of basic block optimizations. Since a single line may be associated with multiple basic blocks, these optimizations need some way to discriminate within a line. Column numbers are an option, but not ideal (see #59612).
ianlancetaylor commentedon Sep 21, 2026
For what it's worth I would expect PGO builds to use DWARF discriminators in the line table. They shouldn't need anything in the runtime traceback information.
zzffeggs commentedon Sep 21, 2026
urandom commentedon Sep 22, 2026
Unless I'm misunderstanding, this is not really true. Consider this real, obfuscated piece of code:
The origin of the panic could be in any of these selectors. You have to start digging to figure out what each type is, and it may still be ambiguous.
It would be interesting to know what the majority of Go's target audience is, and what their constraints are. Most, but not all, of my professional setting did not consider binary size at all. At the same time, it would be quite important in other settings. Since I don't really have the data, I so far made this enabled by default. Do you think it should be opt-in instead?
urandom commentedon Sep 23, 2026
@ianlancetaylor
A followup to my previous comment. Here, with "point of the initial panic", do you mean the innermost frame, and every frame above it is a return after a call?
I'll try to implement another change that only records the columns at sources of panics or other runtime exceptions, to see how the binary size changes.
ianlancetaylor commentedon Sep 24, 2026
Yes.
urandom commentedon Sep 25, 2026
I tested out a a slight variation of the initial change, where a non-zero column is emitted under one of the following conditions:
Tooperand has a symbol name that matches one of the runtime functions that may panic, such asruntime.panicBounds,runtime.panicdottypeI,runtime.growsliceFrom,To, orRestArgsoperands have aTYPE_MEMtype andNAME_NONEname. I think only within this set of operands can there be a fault due to a nil dereference, though I don't really know assembler ...With this, a sample project's size growth fell from 5.2% to 3.4%. For prometheus, the size difference between columns and no columns is 2.9% down from 3.2%. For kubelet, the size difference is 2.7% — this one I did not test before.
There seems to also be another path for detecting the faults, though it doesn't appear to be currently exposed to the linker. In the SSA stage there is a nil checker that seems to have a more precise information on the operands. Various architectures set
faultOnNilArg*for certain opcodes, for operands of TYPE_MEM. I currently do not how that information can be exposed further on, but it might reduce the size difference even further.@ianlancetaylor
How do you suggest I proceed with this? Should I amend the current gerrit change with my latest local one so that others can see them? Should I look into how to use the SSA information?
ianlancetaylor commentedon Sep 26, 2026
@urandom To be clear, I'm just kibbitzing. That said, I'm not clear on something: have you considered only emitting column information when there is a possible ambiguity, such as multiple panic sources and/or multiple function calls on a single line?
komuw commentedon Sep 26, 2026
related #81629
urandom commentedon Sep 26, 2026
@ianlancetaylor
Yes, I did consider that. The way I see it, column that would be most useful if there's at least 2 panic sources on a line. I figure that would further cut down on the storage cost. I haven't written it down, but I figure that for each symbol, I'd have to go over all opcodes, and if there's more than two that would panic, to store the line itself in a map, which I'd then pass to the
pctocolumnfunction. Then a column would be emitted only if opcode being iterated on happens to be on one of those lines, as well as being able to panic. It could add quite a bit of complexity on that path.