Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: heap, allocs, block, and mutex profiles truncate stacks at 32 frames #43669

Open
aclements opened this issue Jan 13, 2021 · 1 comment
Open
Labels
Milestone

Comments

@aclements
Copy link
Member

@aclements aclements commented Jan 13, 2021

The runtime currently (as of Go 1.15) limits recorded stacks of several profile types to 32 frames. This causes deep stacks to become unrooted in pprof call graphs, leading to confusing results and interfering with aggregate statistics up the call graph.

These are hard-coded array bounds in the public runtime API for StackRecord (used in BlockProfileRecord) and MemProfileRecord. Unfortunately, this means these are part of the Go 1 runtime API, so we can't just change them. If we were to increase them, this would impact memory for relatively little gain, since stacks typically aren't this deep. Finally, we probably need some reasonable bound on stack depth just to avoid blowing up profiles.

The "right" fix, in my opinion, is to do what we did with CPU profiles: deprecate the runtime package interface in favor of runtime/pprof, create a log-based back channel from runtime to runtime/pprof that can handle variable-sized stack traces, and perform accumulation in runtime/pprof. That's quite a bit of engineering for a fairly obscure problem, though it would also help with labeling non-CPU profiles (#23458).

We don't have plans to fix this right now. This is just a tracking issue.

@aclements aclements added this to the Unplanned milestone Jan 13, 2021
@prattmic
Copy link
Member

@prattmic prattmic commented Jan 13, 2021

Note for clarity: CPU profiles also have a limit: 64 frames. However, the back channel interface to runtime/pprof means that we could change that limit easily without breaking the API, and the variable size interface avoid extra memory use unless frames are actually that large.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.