Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add abitility to benchmark using CPU time #2

Closed
Shimuuar opened this Issue · 2 comments

4 participants

@Shimuuar

Currently criterion only able to benchmark functions using wall clock time. My experiments show that such measurements are quite sensitive to CPU load and execution time could easily double on heavy loaded system. What's worse such measurements are not reproducible. Every successive run could get different answer and discrepancy couldn't be explained by statistical fluctuations.

I did quick experiment and replaced getPOSIXTime with getCPUTime. In this case measurements didn't depend on CPU load as one could expect. So CPU time is better performance metric for some functions, like numeric ones.

@neothemachine

This should be integrated. It's also common practice in Java to use System.nanoTime() instead of System.currentTimeMillis() for benchmarks as the former uses CPU time instead of wall time.

@coreyoconnor

Hi! I have a patch that uses the clock package to use CPU time.

patch: coreyoconnor@37b1216

I have been using this branch for to verify the performance of bind-marshall (a data marshaling library) I've found that this branch provides more consistent results than the released branch. With the released branch a number of performance measurements would have significant variance in the timings. The use of CPU time from the clock package provided very consistent timing measurements.

There appears to be some assumptions in criterion about the clock accuracy. I find the default estimations of how many samples to collect significantly higher using this patch. Not sure what's going on with this.

@bos bos referenced this issue from a commit
@bos Add support for measuring CPU time
This addresses part of gh-2, although we still need to be able to
perform a regression on the measurement.
734b922
@bos bos closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.