runtime: label runtime functions based on their roles

[hyangah]

As runtime provides more features, we often observe a program's profile is dominated by runtime functions. For example, this is a cpu profile of a program from blog.google.com/profiling-go-programs article (but compiled with go1.9). I highlighted the runtime function frames with purple color. Even for more complex program, we expect at least a quarter of the profiled CPU time to be associated with runtime.

This is useful to analyze performance issues related to runtime. However, often times, it distracts users from focusing on the application logic. On the other hand, Go runtime provides various functionalities (scheduling, GC, map, synchronization, etc), it's still difficult to answer questions like e.g.,

• what's the CPU time spent by GC?
• what's the CPU time spent by map operation?

without knowledge on runtime implementation.

Go 1.9 introduced profiler labels that allows users to label profile samples.
(http://tip.golang.org/doc/go1.9#pprof-labels)
Users can do aggregation or filtering of samples based on the labels (e.g. rpc methods, users, operation mode, etc).

Similarly, I propose to add labels for runtime functions based on the roles of functions to help aggregating and analyzing the cost involving runtime more conveniently. Labels are grouping of runtime functions curated by runtime developers. Implementation-wise, it doesn't necessarily have to use the similar mechanism like the profiler label. Instead, when pprof proto is constructed, it can check the existence of the runtime functions and label based on them.

note 1: Alternative approach discussed was to name the runtime functions based on feature it offers, (e.g. all GC related functions should be prefixed with GC or gc, etc) and users use them to filter or hide using pprof options. That is not as nice as labels, and requires more efforts from users to use it right. Another approach mentioned was to filter out uninteresting frames when publishing or visualizing profiles. That is not ideal because sometimes we want the details and also the list of 'uninteresting' functions may change over time.

note 2: We can envision that pprof recognizes runtime labels and presents the aggregated boxes for each group instead of the detailed runtime call graphs unless asked to display the detailed runtime call graph.

@rakyll @randall77 @aclements

[AlekSi]

That's awesome. Hopefully, someone from Go team will chime in soon.

[bradfitz]

@AlekSi, @hyangah is on the Go team.

[AlekSi]

… Github should add 🤦‍♂️ reaction so I could add it without writing another comment.