Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration #42502

Open
rogerlucena opened this issue Nov 11, 2020 · 15 comments · May be fixed by #42541
Open

runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration #42502

rogerlucena opened this issue Nov 11, 2020 · 15 comments · May be fixed by #42541

Comments

@rogerlucena
Copy link

@rogerlucena rogerlucena commented Nov 11, 2020

I think SetCPUProfileRate is, albeit confusing, still functioning. It needs to be called in the right order with StartCPUProfile. Last time I looked (~a month ago), it probably prints some warnings, but it does change the rate.

The pprof package doesn't seem to have a way to set the profiling rate. Not sure if this is really a useful feature.

Originally posted by @cherrymui in #40094 (comment)

The pprof package indeed does not allow us to customize the CPU profiling rate at the moment. This was something critical for my application in google/badwolf#155, since the default profiling rate of 100 was extremely low and did not allow me no see anything in my CPU profile.

To work around this, I had to make a call to runtime.SetCPUProfileRate(800) (with a rate of around 800 hz to see something meaningful and helpful in my CPU profile) before calling pprof.StartCPUProfile. Since there is a hard-coded call to runtime.SetCPUProfileRate(100) inside the pprof.StartCPUProfile, a warning is being printed to my terminal at the moment saying "runtime: cannot set cpu profile rate until previous profile has finished" as this call to runtime.SetCPUProfileRate(100) failed (the boolean cpuprof.on was already set to true in the previous call to runtime.SetCPUProfileRate(800)). This warning is annoying since it is misleading for someone who does not know the internals of pprof.StartCPUProfile, but the important here is that the CPU profiling is done with the first profiling rate that I manually set (800, as I wanted and as wanted from the user point of view).

Given this, what I am trying to say is that the public/exported method runtime.SetCPUProfileRate is still useful, and sometimes even critical for allowing us to use well pprof and to see something relevant on our profiles, depending on the application. I am not alone on this as you can see here (stack overflow) and here (google groups discussion). Even one principal software engineer (L8) I have been working with at Google found it useful when reviewing my PR google/badwolf#155 above and copied the code to use it in one of his applications, on which setting a higher profiling rate was critical as well.

That said, I believe we could, from now on, follow three different paths:

  1. The ideal would be for the pprof.StartCPUProfile method already receive as argument the CPU profile rate we want, allowing us to customize it, with this argument having its default value set to 100 if it was not specified by the function call. This would be the ideal, if we had support for default arguments in Go. But, we can achieve something similar if we had a similar method with signature pprof.StartCPUProfileWithSpecifiedRate(w io.Writer, hz int), in addition to the pprof.StartCPUProfile(w io.Writer) method we already have. This way, the runtime.SetCPUProfileRate method would not necessarily need to be an exported method anymore.

  2. We could keep runtime.SetCPUProfileRate public/exported, but we would have to fix the problem with the misleading warning message "runtime: cannot set cpu profile rate until previous profile has finished" if we had already manually set the profiling rate before the call to pprof.StartCPUProfile. For this we could:

    2.1. Have an auxiliar boolean method runtime.isCPUProfileRateAlreadySet, and call it inside pprof.StartCPUProfile before the hard-coded call to runtime.SetCPUProfileRate(100) there, trying to set the profiling rate to 100 only if this rate was not already previously set. This way, we would not see any strange warning after a call to pprof.StartCPUProfile if we had already previously set the profiling rate manually. I like this solution because it makes minimal changes on your code, it does not change any signature and it resolves exactly the problem we are trying to solve here, while keeping the warning message for other calls to runtime.SetCPUProfileRate that were not from pprof.StartCPUProfile in the situation above.

    2.2. Make runtime.SetCPUProfileRate return an error in the case the profiling rate was already previously set, allowing the caller to handle it and decide what to do from then on (inside pprof.StartCPUProfile we could ignore this error for example). This would change the signature of runtime.SetCPUProfileRate.

  3. We could move the SetCPUProfileRate public signature to the pprof package (with some changes as explained above to avoid the misleading warning message), and deprecate it / make it unexported inside the runtime package.

These were just some of the alternatives that I thought about for now. I would like to hear more from you on what you think about them and if you have any other suggestions for fixing what I explained above.

@ianlancetaylor ianlancetaylor changed the title Alternatives for the future of runtime.SetCPUProfileRate and pprof.StartCPUProfile runtime/pprof: support setting CPU profiling rate to a value other than 100 Nov 11, 2020
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Nov 11, 2020
imxyb added a commit to imxyb/go that referenced this issue Nov 12, 2020
runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 12, 2020

Change https://golang.org/cl/269437 mentions this issue: runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.

@imxyb
Copy link
Contributor

@imxyb imxyb commented Nov 12, 2020

so we can add method StartCPUProfileWithSpecifiedRate for this issue? @ianlancetaylor

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 12, 2020

I don't think that is the best name, but, yes, something along those lines could work.

@imxyb
Copy link
Contributor

@imxyb imxyb commented Nov 12, 2020

ok, let me think about it

imxyb added a commit to imxyb/go that referenced this issue Nov 12, 2020
runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.
imxyb added a commit to imxyb/go that referenced this issue Nov 12, 2020
runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 12, 2020

Change https://golang.org/cl/269537 mentions this issue: runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.

@ianlancetaylor ianlancetaylor changed the title runtime/pprof: support setting CPU profiling rate to a value other than 100 proposal: runtime/pprof: add StartCPUProfileWithRate Nov 16, 2020
@gopherbot gopherbot added the Proposal label Nov 16, 2020
@ianlancetaylor ianlancetaylor modified the milestones: Backlog, Proposal Nov 16, 2020
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Nov 16, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 16, 2020

I turned this issue into a proposal for the API change.

@hyangah
Copy link
Contributor

@hyangah hyangah commented Nov 16, 2020

@ianlancetaylor Thanks for turning this into a proposal.

The current proposal is specific to StartCPUProfileWithRate. If we decide to add an additional API, can we come up with a version that's flexible enough to be extensible in the future? As far as I remember we had a couple of API change requests before and most of them were rejected or put on hold because of the desire to avoid API changes.

  1. PMU-based profiling: it seems one of the main blockers of the proposal #36821 is currently the API change (in addition to the implementation details). This proposal was originally proposing pprof.StartCPUProfileWithConfig.
  2. Symbolization: I once proposed an option to skip symbolization because the pprof tool may have a way to perform additional symbolization out of band.
  3. Different policy for profiling rate setup: the example issue @rhysh reported (#35057) made me wonder if we may end up with a completely different cpu profiling policies in the future that requires beyond a simple hz adjustment. For example, some users may want to try per-thread timer (#14434) if the runtime supports. Some users may want to adjust the profile rate based on the GOMAXPROCS (#35057).
@rsc
Copy link
Contributor

@rsc rsc commented Nov 18, 2020

@hyangah It sounds like you are suggesting some kind of

cpu := pprof.NewCPUProfile()
... configure prof ...
cpu.Start()

or even

cpu := new(pprof.CPUProfile)
...
cpu.Start()

?

@rsc rsc moved this from Incoming to Active in Proposals Nov 18, 2020
@rsc
Copy link
Contributor

@rsc rsc commented Dec 2, 2020

It sounds like maybe the interface in the previous message is the answer if we want to solve this problem.
But do we need to solve this problem? How often do people need profile rates other than 100 Hz?
The full stack trace makes the profile event a bit expensive, and I wouldn't want people to set the rate very high and then complain about the overhead.

@hyangah
Copy link
Contributor

@hyangah hyangah commented Dec 2, 2020

@rsc Choosing the right profiling rate is hard. There are many theories and different opinion around choosing the right value - http://www.brendangregg.com/blog/2014-06-22/perf-cpu-sample.html is talking about a lockstep sampling issue, #35057 is discussing scaling the frequency based on the system's capability because 100hz is sometimes too high, and @rogerlucena wants higher than 100hz (aside from whether 800 hz is reasonable or not). It looks to me it's a parameter people still want to experiment and tune.

@rogerlucena
Copy link
Author

@rogerlucena rogerlucena commented Dec 7, 2020

But do we need to solve this problem? How often do people need profile rates other than 100 Hz?

@rsc For my application it was crucial to set a sampling rate higher than 100 Hz to see something useful in the profiling output, as I explained and detailed in the Issue description above. Apart from that, in this description I also referenced other people for which customizing the sampling rate can still be useful: stack overflow and google groups discussion.

@rsc
Copy link
Contributor

@rsc rsc commented Dec 9, 2020

OK, well then it sounds like the API in my earlier comment, which introduces the idea of a configurable-because-not-yet-running project, would be good for rate and maybe also leave open a way to handle things like other kinds of performance counters.

#42502 (comment)

Do I have that right? I will retitle this and see what people think.

@rsc rsc changed the title proposal: runtime/pprof: add StartCPUProfileWithRate proposal: runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration Dec 9, 2020
@rogerlucena
Copy link
Author

@rogerlucena rogerlucena commented Dec 10, 2020

Do I have that right? I will retitle this and see what people think.

@rsc lgtm.

@rsc
Copy link
Contributor

@rsc rsc commented Dec 16, 2020

Based on the discussion above, this seems like a likely accept.
(NewCPUProfile + CPUProfile.SetRate + CPUProfile.Start).

@rsc rsc moved this from Active to Likely Accept in Proposals Dec 16, 2020
@rsc
Copy link
Contributor

@rsc rsc commented Jan 6, 2021

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc moved this from Likely Accept to Accepted in Proposals Jan 6, 2021
@rsc rsc changed the title proposal: runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration Jan 6, 2021
@rsc rsc modified the milestones: Proposal, Backlog Jan 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.