Skip to content

Run the garbage collector before each benchmark #10

Merged
merged 1 commit into from Mar 2, 2012

2 participants

@tibbe
tibbe commented Feb 29, 2012

This makes sure that garbage created by previous benchmarks doesn't affect the current benchmark.

Here's an example of running an allocation heavy benchmark, insert/Int, followed by an allocation free benchmark, lookup/Int, prior to my change:

warming up
estimating clock resolution...
mean is 1.704824 us (320001 iterations)
found 1648 outliers among 319999 samples (0.5%)
  1206 (0.4%) high severe
estimating cost of a clock call...
mean is 53.82255 ns (10 iterations)

benchmarking insert/Int
mean: 1.211003 ms, lb 1.124407 ms, ub 1.356176 ms, ci 0.950
std dev: 560.4286 us, lb 343.8345 us, ub 771.3425 us, ci 0.950
found 6 outliers among 100 samples (6.0%)
  6 (6.0%) high severe
variance introduced by outliers: 98.928%
variance is severely inflated by outliers

benchmarking lookup/Int
mean: 194.1801 us, lb 192.9880 us, ub 195.5927 us, ci 0.950
std dev: 6.588454 us, lb 5.524500 us, ub 8.255747 us, ci 0.950
found 5 outliers among 100 samples (5.0%)
  2 (2.0%) high mild
  2 (2.0%) high severe
variance introduced by outliers: 29.696%
variance is moderately inflated by outliers

And after my change:

warming up
estimating clock resolution...
mean is 1.709502 us (320001 iterations)
found 1691 outliers among 319999 samples (0.5%)
  1197 (0.4%) high severe
estimating cost of a clock call...
mean is 51.61498 ns (11 iterations)
found 1 outliers among 11 samples (9.1%)
  1 (9.1%) high mild

benchmarking insert/Int
mean: 1.206301 ms, lb 1.120921 ms, ub 1.352673 ms, ci 0.950
std dev: 557.1362 us, lb 361.4757 us, ub 777.2095 us, ci 0.950
found 6 outliers among 100 samples (6.0%)
  6 (6.0%) high severe
variance introduced by outliers: 98.928%
variance is severely inflated by outliers

benchmarking lookup/Int
mean: 174.1413 us, lb 173.5130 us, ub 174.8956 us, ci 0.950
std dev: 3.503936 us, lb 2.863068 us, ub 4.444968 us, ci 0.950
found 11 outliers among 100 samples (11.0%)
  4 (4.0%) low mild
  5 (5.0%) high mild
  2 (2.0%) high severe
variance introduced by outliers: 13.244%
variance is moderately inflated by outliers

Notice how lookup/Int is now faster, presumably because fewer garbage collections (in particular major ones) happened during the run.

@tibbe tibbe Run the garbage collector before each benchmark
This makes sure that garabage created by previous benchmarks don't
affect the current benchmark.
c868e51
@bos bos merged commit c868e51 into bos:master Mar 2, 2012
@bos
Owner
bos commented Mar 2, 2012

Thanks, Johan!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.