runtime/metrics: CPU metrics update at a relatively low rate

[mknyszek]

Currently they update every GC cycle. It should be straightforward to make them "continuously" updating, since most (all?) of them are accumulated in real time.

[gopherbot]

Change https://go.dev/cl/487215 mentions this issue: runtime/metrics: make CPU stats real-time

[aktau]

What's still needed to have on-demand CPU stats, after the refactoring?

[mknyszek]

The main problem I ran into is monotonicity issues with user time. This arises because user time is computed as the subtraction of everything else from the computed total. For example, if there's a mark assist in flight (but not completed) while we take a measurement, and the take another measurement a small amount of time later (less than the total mark assist time), user time will appear to spike up, then down. More concretely: mark assist takes 10µs. 5µs in we measure. That 5µs on that one CPU, at that time will be attributed to user time. Then, 5µs later, when it's done, we measure again. Now it's attributed to mark assists, so user time may appear to skip backwards.

It's complicated, but we'd need something like https://cs.opensource.google/go/go/+/master:src/runtime/mgclimit.go;l=208;drc=70fbc88288143c218fde9f905a38d55505adfb2b;bpv=1;bpt=1 that's even more complete. Not impossible, but probably a substantial amount of work.

[aktau]

Thanks for the explanation. Before the ideal solution is implemented, which would be complicated as you say, what about an intermediate solution that gets us most of the way there?

• If we can detect that a mark assist is in progress, don't update.
• Optionally, to prevent stat starvation if mark assists can be back-to-back ~forever (don't know if they can), set a bit which forces the next mark assist to update the stat before locking.

In this way, the metric wouldn't be perfectly on-demand, but retain its monotonicity (which is important for metrics) and be less stale than it currently is.

[mknyszek]

To be clear, the mark assist example was just an example (and maybe a bad one; that one we could actually already handle today...). There are several other cases, like the GC workers actively running. It's not super useful to wait until they finish running because that's equivalent to waiting until the end of the GC mark phase.

[aktau]

It's not super useful to wait until they finish running because that's equivalent to waiting until the end of the GC mark phase.

I wouldn't advocate for that. My suggestion was to provide the last snapshot if this is the case:

GCWorker1     ---==----------------------
GCWorker2     --------------====---------
MetricsPoller ---||---------||||----------

Where there is a bar (|), the metrics poller would just see the last updated value. On-demand metrics would start again when no worker is busy updating.