Make runtime benchmark harness more flexible to use

[Kobzol]

Some runtime benchmarks need only read-only access to some input data on which they operate on. Currently, it wasn't easily possible to express this in the runtime benchmark harness, and even though a benchmark only needed to read from some data, the data had to be re-created for each iteration of the benchmark (twice actually, since we run each benchmark two times, to gather instructions and time).

This meant that the data for each benchmark had to be re-created many times, which could slow down the execution of the whole benchmark group. And also the data was always created anew, possibly on a new memory address, which could introduce more noise into the benchmark, because each iteration of the benchmark was operating potentially on different regions of memory.

After this change, benchmarks that only need read-only access to some input data should only create that data once and then read it repeatedly. This can in theory mean that the data will be more available in L1/L2/L3 caches, but that's an orthogonal problem. It is even a question whether we want to avoid this, we could instead warm up the cache and perform measurements on a "prepared cache state", which could further reduce measurement variation. In any case, I have some code prepared to flush the caches if this would be considered a problem.

[Kobzol]

One slightly annoying thing is that the data will be prepared even if the runtime binary is just executed with the list parameter, which returns a list of benchmarks contained in the binary. I'm not sure what to do about that though. We only use the list command for nicer CLI output and duplicate benchmark detection, we could gather the benchmark names only when we actually run the runtime benchmark binary instead..

[nnethercote]

One slightly annoying thing is that the data will be prepared even if the runtime binary is just executed with the list parameter, which returns a list of benchmarks contained in the binary.

Should this be mentioned in a comment somewhere?

[Kobzol]

Thanks, I have expanded the README of the runtime benchmarks directory.