Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sampling scheduler #87

Merged
merged 1 commit into from
Apr 19, 2023
Merged

Conversation

cyhii
Copy link
Contributor

@cyhii cyhii commented Apr 7, 2023

Full profiling is great but sometimes too large for production applications, sampling may help. For example, profiling at a sampling rate 0.1 means profiling for 1s every 10s. The sampling rate could based on PYROSCOPE_UPLOAD_INTERVAL

Supported configs:

  • PYROSCOPE_SAMPLING_DURATION, specify sampling duration directly, must be less than PYROSCOPE_UPLOAD_INTERVAL
  • PYROSCOPE_SAMPLING_RATE, sampling rate, used to calculate sampling duration, multiplied by PYROSCOPE_UPLOAD_INTERVAL. Its value must be in (0, 1) area.

Set one of them should be enough for sampling. If both PYROSCOPE_SAMPLING_DURATION and PYROSCOPE_SAMPLING_RATE are set, PYROSCOPE_SAMPLING_RATE will be ignored.

Not sure this feature is useful or not, need some reviews,

thanks.

Copy link
Collaborator

@korniltsev korniltsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks for the contribution.

just couple of nits: we need to align profiling time to 10s interval and mark it as an experimental so we do not commit to support it forever and a bit more flexible in the future

}
profiler.stop();

Snapshot snapshot = profiler.dumpProfile(profilingStartTime);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need profilingStartTime to be truncated to 10s(uploadInterval), see #40

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to truncate(profilingStartTime, uploadInterval)

import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

public class SamplingProfilingScheduler implements ProfilingScheduler {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would maybe mark this class and corresponding configuration options as experimental ( for example just add comment in javadoc of the class)
This way we'll be able to change API in the future, for example someone may wish to do profiling for 10s or only when the moon is full or some other sophisticated schedule

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments added for class SamplingProfilingScheduler, config string PYROSCOPE_SAMPLING_RATE and PYROSCOPE_SAMPLING_DURATION, please check if it is ok.

@korniltsev
Copy link
Collaborator

Also I wonder if you really need it to be in the pyroscope-java or maybe you could jsut implement it in your app and plug it in with our builder api?

@cyhii
Copy link
Contributor Author

cyhii commented Apr 11, 2023

Builder API is quite wonderful, but in some cases we want to use pyroscope just as a java agent, so it seems to be a little hard to customize the sampling way if there is no configuration support.

I do missed the aligning feature, thanks for your review, I will adjust my code later.

Copy link
Collaborator

@korniltsev korniltsev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

I am a bit concerned that profiling end time is not correctly reported - it is startTime + uploadInterval, while here it should be startTime + uploadInterval*samplingRate or smth. Another thing that may be slightly incorrect is a sampleRate / total time during rendering - if a single profile reports 1s cpu time - it will show 1s cpu time in ui, but in reality it may have been 10s. I guess we can deal with it later if this becomes a problem.

@cyhii
Copy link
Contributor Author

cyhii commented Apr 17, 2023

It's OK. Thanks for you reviews.

@korniltsev korniltsev merged commit 3fc20fa into grafana:main Apr 19, 2023
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants