Fix Windows resolution problems by replacing Data.Time.Clock.POSIX with System.Clock

[conklech]

On Windows, getPOSIXTime uses the C function GetSystemTimeAsFileTime, which gives low-resolution timings. As observed in issue #11, it will return the same value many times in a row. criterion therefore produces low-quality statistics under Windows. QueryPerformanceCounter is the appropriate timing source for benchmarks on Windows; we can access it with Clock.getTime Clock.Monotonic from the clock package. See example data below.

My patch, conklech/criterion@b6f657c, fixes the problem on Windows, but might create compatibility problems on some POSIX systems.

In issue #2, @coreyoconnor gave a similar patch, coreyoconnor/criterion@37b1216, which uses Clock.ProcessCPUTime instead. That has different semantics--which was the goal there. But it's not clear that Clock.ProcessCPUTime gives benchmark-grade data on Windows. It uses GetProcessTimes under the hood; MSDN doesn't explain where the data comes from, and I haven't tested it.

Under POSIX, time calls gettimeofday and clock calls clock_gettime. Cursory Google research indicates that clock_gettime with CLOCK_MONOTONIC is the proper benchmark timesource; gettimeofday can go backwards, for instance when ntp is adjusting the clock.

However, it looks like some systems don't implement CLOCK_MONOTONIC. Perhaps just OS X? I don't know. Someone with better POSIX knowledge should look into compatibility issues with CLOCK_MONOTONIC in non-Win32 environments.

On my Windows 7 laptop, criterion using time (i.e. criterion-0.6.2.1) reports an estimated resolution of about 3.2 msec, with the weird 200% outlier rate reported by others. However, the final report data shows that all measurements share the same two or three values:

After dropping in clock and using Clock.Monotonic, the resolution is reported as 4.5 msec but the outlier rate problem is gone, and the final report shows an actual distribution of data:

[mikesteele81]

I'm using Windows 7 64-bit. With this patch I see a similar improvement.

[Toxaris]

What's the status of this? Is unpatched criterion good for benchmarking on Windows by now?

[bos]

Thanks, I've fixed this in 53d3203.

The fix is fairly involved because it needs to work on multiple platforms. I particularly commend you to read the comment at the start of the Windows file.

[mikesteele81]

Thanks for the update. I'll use criterion within Windows more often, and see how this improves time measurements. It looks like the clock library does things similarly.

At first I thought calling QueryPerformanceFrequency once at the start of testing was the wrong way to go about it. I seem to remember a Microsoft example that called it once before every pair of calls to QueryPerformanceCounter, which indicated to me the performance counter's frequency was not stable.

According to a comment on QueryPerformanceFrequency, the behavior is affected by which timing hardware is available to Windows. Perhaps modern computers all have a stable "TimeStampCounter".

[dreixel]

I still see this bug with criterion-0.8.0.1. Is the patch not yet in the latest released version?

[bos]

There is not a release that has this fix. You'll want to use HEAD for a bit if you need it.

[dreixel]

Ok, thanks.