proposal: runtime/pprof: option to profile without symbolization

[hyangah]

We propose to extend runtime/pprof API to allow users to choose whether to
include symbol info in the profile output.

Background

Since Go1.9, profile outputs include symbol information
(https://golang.org/doc/go1.9#go-tool-pprof) so they can be interpreted
without the binary. Symbolization is done by the profiled binary and we
observed in some cases, such symbolization work in the profiled target
binary is not desirable.

For example,

• we may want to avoid adding extra load to a busy server when
  debugging CPU overload issue.
• symbol information may be available in different sources (binary, external
  symbolization service, etc) and users may favor to use them instead.
• currently runtime delegates symbolization of non-Go symbols to cgoSymbolizer and its efficiency
  is sometimes not easy to control.

The pprof tool still supports retrieving symbol information from external sources
when it finds the fetched profile lacks symbol information.

API changes

runtime/pprof: In the scope of this proposal we care only about controlling
symbolization, but one can imagine that we may want to support more customization
in profiling (for example, filtering, choosing different output formats, debug levels,
changing CPU profiling rate, etc). In order to support the future extension easily, we
propose to add profile parameter types.

Then, StartCPUProfile and Profile.WriteTo are variations that take those new parameter
types.

package pprof 
// CPUProfileParam is a set of parameters to configure CPU profiling.
type CPUProfileParam struct {
  ProfileParam
  // Place for future CPU profile-specific parameters
}

// ProfileParam is a set of parameters to configure profile output generation.
type ProfileParam struct {
  NoSymbol bool
}

// StartCPUProfileWithParam is like StartCPUProfile but it allow to configure
// profiling and its output generation.
func StartCPUProfileWithParam(w io.Writer, param CPUProfileParam) error

// WriteToWithParam is like WriteTo, but it allows to configure profile output generation.
func (* Profile) WriteToWithParam(w io.Writer, param ProfileParam) error

net/http/pprof:

Profile handlers can take a new 'nosymbol' form value. If it is set to non-zero integer
value, profile output will not include symbol information.

There will be no change affecting existing API users.

Implementation

Implementation is trivial - we skip symbolization that had been done while formatting
the output. The pprof tool can perform symbolization correctly as long as the Location
entries of the output proto contain correct Address and Mapping information.

One caveat is that currently expansion of inlined frames are also handled by
runtime.CallersFrames used for symbolization. There is no way to perform expansion
without symbolization. Once the correct DWARF debug information for inlined frames
work (in progress targetting for 1.10) is done, the DWARF reader can expand them
using the PCs.

[gopherbot]

Change https://golang.org/cl/70870 mentions this issue: runtime/pprof: allow cpu profiling without symbolization

[aalexand]

@hyangah https://golang.org/cl/70870 does not yet add WriteToWithParam, is that to be added in that CL?

[aalexand]

@hyangah I can see now the CL says there will be a separate CL.

[rsc]

I would rather just always symbolize and not worry about this. If there are efficiency bugs let's fix those separately.

[hyangah]

@rsc The efficiency issue we are concerning about is more in the foreign language side called by cgoSymbolizer. It will be much easier and more accessible for users if an option to delegate symbolization to pprof is given than requiring them to optimize their C++ or other foreign language libraries sufficiently so not to interfere with Go's profiling code.

When the server is already over-loaded, I'd rather want to minimize work on the server side, and reduce the chance of failed profile collection due to extra operations that can be done offline.

[gopherbot]

Change https://golang.org/cl/76211 mentions this issue: runtime: new Mode field in cgo symbolizer function arg

[rsc]

Per discussion on CL, we'd like to fix any performance problems before resorting to another mode and recreating the old "you have a meaningless profile because it has no symbol info and your binary is gone" failure mode.

[hyangah]

The pprof tool performs symbolization immediately after collecting the profile from a running server and tries its best to produce a profile output with symbol info in it. So, the output is always carrying the symbol lookup if the server's symbolz (debug/pprof/symbol) responds to the symbol lookup requests.

I still feel that we need to distinguish the case where we get the profile using the 'go test' or through the current runtime/pprof API, from the case where we get the profile from the live server.

Unfortunately, I just noticed that the symbolz or debug/pprof/symbol handler is implemented with runtime.FuncForPC which is incomplete compared to runtime.CallersFrames (missing inline function info, and also not retrieving non-go symbols).

[rsc]

@hyangah, have you been able to optimized the C++ symbol lookup instead? Can we close this issue?