runtime/trace, cmd/trace: provide a per-M view

[prattmic]

The Go execution tracer currently provides a per-P view (default, via /trace) or a per-G view (via /goroutines).

When investigating issues with the scheduler I've frequently found myself wanting a per-M view, as that is the level that the scheduler truly operates on. This makes it easier to see when Ps switch between Ms, when Ms stop, etc.

For #43997, I found the need compelling enough that I built a one-off CL for it in http://golang.org/cl/286972. That CL adds an argument to (almost) every trace type, so I don't think it is acceptable as a general solution, but it would be nice to have a more complete version we could include for real.

cc @mknyszek @aclements @hyangah

[gopherbot]

Change https://golang.org/cl/286972 mentions this issue: runtime,cmd/trace: add M id to traces

[zhouguangyuan0718]

Sorry for bother. Is this feature will be accepted in the future?
@prattmic

[prattmic]

Is this feature will be accepted in the future?

I don't expect much pushback on this feature, however my CL https://golang.org/cl/286972 adds data to the trace file, thus potentially causing a performance impact, meaning it needs more thought, and no one is working on this. If my CL was free, then it likely would already have gone in.

[prattmic]

As a bit of an extension of this, I also dream of extending the trace files in per-M view for "runtime-internals view". I want a complete breakdown of what Ms are doing. i.e., from the moment an M wakes out of mPark until it sleeps again in mPark I want a fully specified trace that shows what that M is doing: looking for work, running a goroutine, etc. Time in usleep or waiting on a lock may be blank, but should be annotated. wakep, handoffp, etc should flow events to the M they wake.

[aclements]

I've been wondering for a while if having a single type of trace is too blunt of an instrument. One could imagine still having a single trace format, but introducing finer control into what sorts of trace events are captured. There's the potential for some unification here, too, for example with the profile stream and with debuglog.