Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Possible error in data analysis #3

Shimuuar opened this Issue Oct 22, 2011 · 1 comment


None yet
2 participants

Shimuuar commented Oct 22, 2011

I think there is an error in criterion's data analysis. Here is simplified description of algorithm as I understand it. For simplicity I will ignore discreteness of clock.

  1. Clock call cost is estimated.
  2. For every benchmark N samples are collected
  3. These samples are corrected for timing overhead and somehow averaged. Exact nature of averaging is not significant for reasons stated below.

The problem is that clock call cost is measured quantity and this measurement have error (σ(t) from now on) and it's never taken into account. This error corresponds to shifts of timing distribution as whole and couldn't be eliminated. No averaging procedure could detect such shifts. So error for benchmark couldn't be less than σ(t). It shouldn't be significant for long-running functions. But it is significant for function which take same or less time to complete as getPOSIXTime. mwc-random's benchmarks should be affected.


bos commented Jul 23, 2014

I've dropped this approach entirely in recent commits.

@bos bos closed this Jul 23, 2014

rrnewton pushed a commit to rrnewton/criterion that referenced this issue Sep 6, 2014

Merge pull request #3 from Shimuuar/master
Reworked type classes for distributions' statistics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment