Add abitility to benchmark using CPU time #2

Shimuuar opened this Issue Oct 18, 2011 · 2 comments


None yet
4 participants

Shimuuar commented Oct 18, 2011

Currently criterion only able to benchmark functions using wall clock time. My experiments show that such measurements are quite sensitive to CPU load and execution time could easily double on heavy loaded system. What's worse such measurements are not reproducible. Every successive run could get different answer and discrepancy couldn't be explained by statistical fluctuations.

I did quick experiment and replaced getPOSIXTime with getCPUTime. In this case measurements didn't depend on CPU load as one could expect. So CPU time is better performance metric for some functions, like numeric ones.

letmaik commented Aug 26, 2012

This should be integrated. It's also common practice in Java to use System.nanoTime() instead of System.currentTimeMillis() for benchmarks as the former uses CPU time instead of wall time.

Hi! I have a patch that uses the clock package to use CPU time.

patch: coreyoconnor/criterion@37b1216

I have been using this branch for to verify the performance of bind-marshall (a data marshaling library) I've found that this branch provides more consistent results than the released branch. With the released branch a number of performance measurements would have significant variance in the timings. The use of CPU time from the clock package provided very consistent timing measurements.

There appears to be some assumptions in criterion about the clock accuracy. I find the default estimations of how many samples to collect significantly higher using this patch. Not sure what's going on with this.

@bos bos added a commit that referenced this issue Jul 26, 2014

@bos bos Add support for measuring CPU time
This addresses part of gh-2, although we still need to be able to
perform a regression on the measurement.

bos closed this Jul 26, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment