Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: support efficient accumulation of custom event count profiles #18454

Open
josharian opened this issue Dec 28, 2016 · 7 comments

Comments

@josharian
Copy link
Contributor

commented Dec 28, 2016

I want to be able to gather pprof-esque information about instantaneous events that have occurred over the lifetime of my program, possibly with sampling for performance reasons.

This is similar to what the heap profile does, at least when used with alloc_space and alloc_objects--it tracks memory allocations over a long period.

The existing runtime/pprof custom profile API seems ill-suited to this. (Insofar as I understand it. See #18453.) One could accomplish it by inventing a unique key for Add and never Remove anything. However, this will result in a giant, ever-growing map. It would be far more efficient to just keep a counter per pc, as many of the runtime-provided profiles do. It might be worth considering adding a different kind of custom profile geared more towards this use case.

I don't have a concrete proposal, since I haven't thought about this deeply. This issue is just intended to open discussion, particularly since pprof labels are coming for 1.9, and it might be worth considering how they interact with custom profiles--hopefully productively.

cc @matloob

@josharian josharian added the Proposal label Dec 28, 2016

@josharian josharian added this to the Proposal milestone Dec 28, 2016

@josharian josharian changed the title proposal: runtime/pprof: support efficient accumulative custom profiles proposal: runtime/pprof: support efficient accumulation custom profiles Dec 28, 2016

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 9, 2017

@josharian Something like

// SetSampleRate sets the profile to record samples randomly with probability 1/n.
// It must be called before any calls to Add, Remove, or Event.
func (*Profile) SetSampleRate(n int)

// Event records that 'weight' events happened.
// For a simple event counter, weight should be 1.
// If events have different expenses, weight can vary according to the expense.
// A given Profile should be populated using Add/Remove or Event, but not both.
func (*Profile) Event(weight float64)

?

The comment text is bad but you get the idea. This would capture the kinds of things that we do for the mutex profile as well as basic counters.

I'd like to make sure we get the other pprof changes through first, but this seems like a reasonable followup.

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented Jan 9, 2017

@rsc yes! SGTM.

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented Jan 30, 2017

Opinions about implementing this using a probabilistic data structure like count-min-sketch-based top-k?

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2017

Ping for opinions. (Or if someone else is going to implement this, that's great, I'll just wait patiently.)

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2017

@josharian Sorry, looks like I dropped a bunch of Github notifications about two weeks ago.

You have more context here than I do, but I don't think a new data structure is required for the API I sketched. There's nothing about "top N" in the usual profiles; it's supposed to be a representative sample of the overall behavior, not just the "weighty" behavior.

I think all that is needed is a func shouldSample(rate int, weight float64) bool that determines whether the event is added to the profile (and then never removed). I believe that code (if not that precise function) already exists for deciding whether to sample a memory allocation. Then the profile itself is a map[stack]float64. I don't know that we have one of those in runtime/pprof right now, but we will once my pending work is done (moving that piece from the runtime to runtime/pprof).

@rsc rsc changed the title proposal: runtime/pprof: support efficient accumulation custom profiles runtime/pprof: support efficient accumulation of custom event count profiles Feb 13, 2017

@rsc rsc added Proposal-Accepted and removed Proposal labels Feb 13, 2017

@rsc rsc modified the milestones: Go1.9, Proposal Feb 13, 2017

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2017

Hmm. I'll look again once your pending work is done.

@Sajmani

This comment has been minimized.

Copy link
Contributor

commented Jun 29, 2018

@josharian Perhaps the code in x/time/rate might be useful for this:
https://godoc.org/golang.org/x/time/rate#NewLimiter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.