New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime/metrics: add goroutine state counts, total goroutines created, total threads #15490
Comments
You can get all of these statistics by parsing the tracing output from
runtime/trace.
|
@minux, while true, runtime/trace seems like a pretty high overhead way to collect what amounts to a fairly small amount of information. It's certainly low overhead for what it does, but what it does is much more than what's needed here. The metrics @deft-code wants are primarily intended for continuous monitoring (based on offline conversations), so it needs to be cheap. |
Here are the notes on the desired metrics I had from our meeting a while ago: Ring buffer of sampled duration between entering and exiting runnable state
Four global stats
Maybe current number of running goroutines |
Assigning to @aclements to decide what we're willing to support long-term. |
Ping @aclements |
Still up to @aclements. |
Ping @aclements. Can you look at this during the release candidate quiet? |
@deft-code, do the specific stats I suggested in #15490 (comment) address your needs? |
Sorry, I'd lost track of the fact that there was a concrete proposal doc for this: https://github.com/deft-code/proposal/blob/master/design/15490-schedstats.md @deft-code, could you mail a CL to add this to the go-proposal repository and, once submitted, edit your first post to link to it? Thanks. |
I'll get on top of it. |
Thanks! |
CL https://golang.org/cl/38180 mentions this issue. |
Do we need to keep this issue open, or should we accept it? |
There's definitely still work to do on how and what exactly the API should expose, but I think it's pretty clear we need to provide some visibility into the scheduler. |
Teams inside Google are patching in CL 38180 and getting some experience with it. If others would like to do the same, please do. We'll probably wait until Go 1.10 to decide to add the API officially. Putting the proposal on hold until then. |
We are looking at ways of detecting overload in CockroachDB, and statistics like described in #15490 (comment) or in the proposal would be extremely helpful. Is there any hope of getting something like this in Go any time soon? I can help with the work if folks are in agreement on what to build. As a timid first step, I was thinking of adding a Also, is there any more information about the experiments to use the number of runnable goroutines as an overload indicator? I understand the fluctuations are an issue, but wouldn't sampling this info frequently and looking at an average over a reasonable timeframe address that? |
Side-stepping whether or not we want this, but now that we have a runtime metrics API (#37112), if we do add these kinds of metrics, adding to the metric API will be the obvious place rather than a new SchedStats method/struct. (e.g., the metric @RaduBerinde proposes above). cc @mknyszek |
@golang/runtime @mknyszek Is there anything here that is not covered by the runtime metrics package? |
I think a bunch of the metrics in https://go.dev/cl/38180 are not covered by the runtime/metrics package. I've had it on my TODO list for a while but it's never quite made it to the top. Clearly not for this release, but I continue to hope. In terms of the proposal process, I don't think this needs to be on hold anymore. Adding new metrics is a fair bit more lightweight than it used to be, so even if we don't see an obvious use-case right now, the bar is low enough that I'm comfortable with just adding the remaining metrics. In terms of the original proposal, I think all we're missing is counts of goroutines in various states, total count of goroutines created (to create a rate metric), and a thread count. (We already have a histogram metric for time spent in "runnable.") |
If someone would like to take a stab at implementing this, please be my guest. Otherwise, I'll get to it next cycle. |
If someone would like to take a stab at implementing this, please be my guest. Otherwise, I'll get to it next cycle. |
@mknyszek Thanks. Should we keep this issue open and retarget to runtime/metrics? Or should we open a new proposal? |
We can keep this issue open and retarget it. I'll update the header and such. |
This proposal has been added to the active column of the proposals project |
Retitled based on #15490 (comment). Have all concerns about this proposal been addressed? |
Yeah, I believe all concerns are addressed. |
Based on the discussion above, this proposal seems like a likely accept. |
No change in consensus, so accepted. 🎉 |
As for what these metrics should be named, perhaps:
The goroutine state metrics come from https://go-review.googlesource.com/c/go/+/38180/9/src/runtime/pstats.go#18. I figure we can reuse most of that implementation. |
|
This issue is currently labeled as early-in-cycle for Go 1.22. |
Update, Jun 7 2023: runtime/metrics now exists, but there are a few metrics in the draft CL here that aren't yet exposed. See #15490 (comment).
MemStats provides a way to monitor allocation and garbage collection.
We need a similar facility to monitor the Scheduler.
Briefly:
The text was updated successfully, but these errors were encountered: