runtime/trace: don't omit runtime (internal) stack frames

[aktau]

Go version

go version go1.26-dev

Output of go env in your module/workspace:

N/A

What did you do?

Recently, I was looking at a trace with @mknyszek and @prattmic when we tried to see where a specific goroutine could be "stuck". When collecting a runtime trace, sometimes stack frames are omitted as they are considered runtime-internal. However, in a number of cases it would be useful to see them.

What did you see happen?

Runtime internal frames were omitted (or we thought they were omitted, but couldn't be sure).

What did you expect to see?

Given that runtime/trace is an expert tool (and often used for runtime investigations), I believe it makes sense to not filter at all. @mknyszek mentioned that this may even simplify the code. (There are traces internally that apparently "fail to walk", likely they are cgo, omitting complexity here may help us get to the bottom of that problem.)

(The same behaviour exists for the stacks generated by runtime.Stack() et al., but it might warrant a separate discussion.)

TODO:

• Find original rationale for omitting runtime frames.
• Find/file issues for other stack interfaces (like runtime.Stack).
  • runtime/pprof: mechanism to show runtime frames in heap profiles #71175 (thanks Gaby)
  • proposal: runtime/pprof: cross system stack transitions in the heap profiler #66385 (thanks Gaby)

[gabyhelp]

Related Issues

• runtime: stack should indicate when stack frames are omitted #53349 (closed)
• runtime/pprof: mechanism to show runtime frames in heap profiles #71175
• runtime/trace: missing stacks for trace events emitted when `getg().mp.curg == nil`, like `GoCreate` for `time.AfterFunc` #68093
• runtime: trace2: some stacks don't skip tracer's own frames #64862
• proposal: runtime/pprof: cross system stack transitions in the heap profiler #66385
• cmd/trace: show end stack traces #70570 (closed)
• proposal: runtime: expose internal/trace package #38971 (closed)
• runtime/race: use runtime.CallersFrames #19749 (closed)
• runtime/pprof: no API to access deep stacktraces with MemProfileRecord and BlockProfileRecord #67941
• runtime/trace: no call stack recorded the first time a goroutine is seen in a trace #65634 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

I don’t know why I wrote #71175 specifically about heap profiles instead of all diagnostics.

[nsrip-dd]

I guess compared to the heap profiler, where there is post-processing to hide the runtime frames, for the tracer it's more a matter of skipping too much? #68093 seems relevant here (edit: ah, I see that Gaby linked it). The tracer currently collects "user" goroutine stacks, i.e. if the event happens on the system stack portion of a user -> system call. So there are some cases where we currently get nothing like in that issue. And even if there is a user stack, we may want to see the system portion, too?

[randall77]

Does setting GOTRACEBACK=system fix it?

[aktau]

I don’t know why I wrote #71175 specifically about heap profiles instead of all diagnostics.

Perhaps: it may be easier to change one interface over the other, and we should take what we can easily get first :).

Does setting GOTRACEBACK=system fix it?

I don't believe it does (for the tracer). But I haven't checked yet.

(Even if it did, I think we would prefer either changing the default or providing a separate knob so we don't globally get some of the other side effects of GOTRACEBACK=system.)