Support for repeated runs

[mre]

Currently we run a benchmark only once for each step. We should repeat the benchmark several times in order to get more accurate results.

There are many ways to calculate the benchmark result for repeated runs:

• Take the average time of all runs
• Report the median time of all runs
• Report best of n (e.g. best-of-three)
• ...

Currently I think about median time as it is a simple way to get rid of outliers.

[markuspoerschke]

+1 Nice. Also want this possibility!

[markuspoerschke]

We need to think about some „problems“.

• how can we detect peaks at the beginning of a function?
• do we repeat each function call with the specific iteration number or repeat the whole bench?
• how do we integrate live rendering of the graph?
• how do we remove "random" peaks

Possible solution:

• gather all graph data at once and replot after n iterations (not every to save some time)
• repeat after a full run of a benchmark
• update the graph data step by step
• this will print the graph like it is printed now. in the iteration you will be able to see how the graph will be "optimized".
• we could create an interface that will receive the raw data and will then return the graph data
• this can also help us to add some more export functionality for example export CSV files instead of images etc.

[mre]

Yes, I agree. We need to separate the data from the graphing.
This way we can have a graph output, an image output, a table output and maybe something even more fancy.
I propose that we start with storing the benchmark data in a separate datastructure. The data can then be used in any way (e.g. by GnuPlot).
After that we can think of good data transformations (filtering, averaging, interpolating and so on).
Finally I want to have an estimation of the big O notation for each graph.

[mre]

Maybe something like this would work.
Most of these settings have sane defaults, so an end-user might probably not need to adjust them.

$phpench = new PHPench('Compare array_flip and array_unique')
              ->addTest(new TestArrayFlip, 'array_flip')
              ->addTest(new TestArrayUnique, 'array_unique')
              ->setInput(range(1, pow(2,16), 1024))
              ->setRepetitions(5)
              ->setAggregator(new MedianAggregator)
              ->setOutput(new GnuPlot)
              ->run()
              ->save('test.png', 1024, 768);

What do you think?

[markuspoerschke]

Sounds good to me.

[markuspoerschke]

Also added MedianAggregator: ec6341a

[mre]

👍

[markuspoerschke]

I think we can close this issue. But we should create a new issue regarding of the output handling.