Currently criterion only able to benchmark functions using wall clock time. My experiments show that such measurements are quite sensitive to CPU load and execution time could easily double on heavy loaded system. What's worse such measurements are not reproducible. Every successive run could get different answer and discrepancy couldn't be explained by statistical fluctuations.
I did quick experiment and replaced getPOSIXTime with getCPUTime. In this case measurements didn't depend on CPU load as one could expect. So CPU time is better performance metric for some functions, like numeric ones.
This should be integrated. It's also common practice in Java to use System.nanoTime() instead of System.currentTimeMillis() for benchmarks as the former uses CPU time instead of wall time.
Hi! I have a patch that uses the clock package to use CPU time.
I have been using this branch for to verify the performance of bind-marshall (a data marshaling library) I've found that this branch provides more consistent results than the released branch. With the released branch a number of performance measurements would have significant variance in the timings. The use of CPU time from the clock package provided very consistent timing measurements.
There appears to be some assumptions in criterion about the clock accuracy. I find the default estimations of how many samples to collect significantly higher using this patch. Not sure what's going on with this.
Add support for measuring CPU time
This addresses part of gh-2, although we still need to be able to
perform a regression on the measurement.