Virtual getter for Benchmark.InnerIterationCount adds overhead

[AndyAyersMS]

The virtual call to Benchmark.get_InnerIterationCount can be on the same order of magnitude as simple operations one might like to time. Since the recommended usage is

            for (int i=0; i<Benchmark.InnerIterationCount; i++)
                // test code here

we are adding one virtual call's worth of overhead per inner iteration and all of this overhead gets timed.

Ideally the getter should be non-virtual. Marking BenchmarkIteratorImpl as final might also work in more recent builds as the getter should then devirtualize.

Alternatively, we should update the recommended usage to hoist the call out of the loop.

[jorive]

Thanks @AndyAyersMS! I'll get to it soon and send a PR.
-José

[adamsitnik]

@AndyAyersMS have you tried BenchmarkDotNet? Our primary feature is to allow the users to have accurate nano-benchmarks. I don't mean to be rude, but I think that xunit-performance is simple and good for micro-benchmarks, but it was not designed for performing accurate nano-benchmarks.

[AndyAyersMS]

@adamsitnik I haven't looked at BenchmarkDotNet yet, no. It is on my todo list.

We are mainly interested in detecting changes (regressions or improvements) in performance, especially over time. So we aren't overly concerned if a test has some overhead, so long as the overhead is consistent over time and is not drowning out the thing we are trying to monitor. Less overhead is always preferable, but some overhead is acceptable.

I'd be curious to know if anyone has studied the durability of BenchmarkDotNet performance numbers over time. We typically see pretty tight clustering of data on an individual run. So our measurement techniques generally seem locally reliable (with some exceptions). But over time spans of days to weeks, we sometimes see shifts where the performance level moves to some new stable value, and in many of these cases we believe nothing significant has changed from one time span to the next.

The working hypothesis is that some benchmarks are critically sensitive to one or more microarchitectural features, and so seemingly irrelevant and perhaps unnoticed changes in how the tests run can dramatically alter the measured performance. For instance, something like heap randomization or ASLR shifts the relative location of branches and so alters the behavior of the branch predictor, or the lack of loop top alignment in the JIT alters how loops interact with various CPU frontend caches.

For example:

Does BenchmarkDotNet have any techniques to mitigate this kind of thing? For example the kinds of on the fly randomization described in stabilizer?

We have enough measurement traffic that we can't do this all our measurements on one machine. So we have a set of matched machines we use. I have done some initial studies looking for test-machine correlations, and haven't spotted any, though it is tricky to sort this out from the other fluctuations since we haven't root caused all of those. Has anyone looked at how stable the BenchmarkDotNet numbers are when running on different -- but very similar -- machines?

[AndreyAkinshin]

Hey @AndyAyersMS.

BenchmarkDotNet has a lot of techniques which provides stable results if you are running a benchmark on the same machine all the time. It automatically chooses an amount of iteration (e.g. if a 3rd party Windows process eats some CPU at the middle of the benchmark and spoil the average; the amount of iteration will be increased), handles the outliers, prints a lot of statistics, and so on. Also, it knows about many microarchitectural "features" (like different branch prediction strategies on Haswell+) which can be a cause of unstable results; and BenchmarkDotNet knows how to handle it.

The working hypothesis is that some benchmarks are critically sensitive to one or more microarchitectural features,

Yeah, you are absolutely correct, it can be a huge problem for microbenchmarks.

different -- but very similar -- machines

There is no such thing like "very similar machines" in the world of microperformance. =) Could you tell me the exact machine configuration? Sometimes even minor configuration differences could play the main role in the final performance.

we sometimes see shifts where the performance level moves to some new stable value

It's wrong to put only average values on the plot because we have to compare distributions instead of single values. We can observe different "stable values" because of different reasons, but we can't say that there is a significant difference if we don't know the whole distribution: we need the confidence interval, we need the standard deviation, and so on.

Does BenchmarkDotNet have any techniques to mitigate this kind of thing? For example the kinds of on the fly randomization described in stabilizer?

BenchmarkDotNet has some of the techniques. E.g. it introduces dummy methods around the target benchmark with many increment instructions which allow decreasing the branch instruction density (it seems that Intel CPUs has some heuristics which use this density for some of the microoptimizations).
BenchmarkDotNet doesn't use any randomization techniques for now, but we can try it and check out how it will work in real environments.

---

Conclusion. Primary, BenchmarkDotNet is a tool for local performance investigations. And I don't think that it's possible to create a general tool which allows running performance tests on different machines and always get the consistent results. If you have two different machines, it's almost always possible to write such benchmark which will show different results on these machines.
However, if we will not write such tricky benchmarks on purpose, it seems to be possible to handle the most popular problem. BenchmarkDotNet already has a gentleman's set for performance investigations. Probably, we will never cover all the problems in the world, but we can try to investigate performance phenomena one by one, fix them somehow and decrease the percent of "false performance degradation alarms."
If you are going to try BenchmarkDotNet for your perf tests, I would love to help with these investigations and implement some advanced techniques for stabilizations of the results.

[AndyAyersMS]

@AndreyAkinshin, I would definitely like to understand some of the techniques you've developed better. Sounds like I should rearrange my todo list.

As a practical matter we have to run tests on multiple machines. There's just too much that needs to be measured. So we have to deal with this up front. @valenis can describe what we do to ensure the machines are as identical as possible, if you're curious.

I haven't said much about what data we record or how we analyze it. We keep track of individual iteration results (these are typically aggregations of many runs of some target method) and record and consider distributions.

We haven't gotten to the point where we do any sort of automated inference or regression detection. I am working on some tooling for this now. Likely it will be non-parametric as most of our distributions are far from normal and sample sizes are small. But it is pretty evident from the data that the "multiple stable" level phenomena is real and the distribution really does shift over time.

It's probably worth putting together an experiment where we take some of our troublesome benchmarks and reharness them under BenchmarkDotNet and see how stably they perform over time. A week or two where we run each test once per hour should be enough.

Note we're not just running the same test over and over, we are testing each successive commit's build. For most commits little or nothing will have changed from the previous build and so we don't expect performance to change very often. Yet it seemingly does, for some tests.

All our tests are out in the open here if you feel like pitching in. dotnet/coreclr#11653 has a rough noise level estimate per test; dotnet/coreclr#11657 shows tests that have exhibited multiple stable levels over time.

[AndreyAkinshin]

It's probably worth putting together an experiment where we take some of our troublesome benchmarks and reharness them under BenchmarkDotNet and see how stably they perform over time.

Sounds awesome. Let me know if you need any help with BenchmarkDotNet.

@valenis can describe what we do to ensure the machines are as identical as possible, if you're curious.
All our tests are out in the open here if you feel like pitching in. dotnet/coreclr#11653 has a rough noise level estimate per test; dotnet/coreclr#11657 shows tests that have exhibited multiple stable levels over time.

I like performance puzzles and I would like to check what's going on in these benchmarks. Is it possible to get the full measured data and machine configurations for each run?

[AndyAyersMS]

I can get you selected data sets, sure. I have machine IDs for each run but not configurations; it may take me a day or two to add the HW details in.

Suggest we start with Span / SpanBench / TestSpanFillByte(length: 1000); here's a sketch of what it has done over the past 2 months:

If that sounds good, I will post the data set on a gist in CSV sometime later today, and we can work up a BenchmarkDotNet version to use as a test subject.

[AndreyAkinshin]

If that sounds good, I will post the data set on a gist in CSV sometime later today, and we can work up a BenchmarkDotNet version to use as a test subject.

LGTM, waiting for CSV (don't forget about machine IDs).

[AndyAyersMS]

Here's the CSV: https://gist.github.com/AndyAyersMS/e981f5540f1e1d7966279dc0c6ccfaf4

About 140 executions of the test over a period of roughly a month. BuildId is the matching CoreCLR commit hash. The data here matches the blue "Duration" lines above (though the CSV starts Apr 1 so is missing the first few runs on the left).

As far as I know nothing deliberate was done that would have affected perf of this test during this time.

[AndreyAkinshin]

It looks very strange, something is definitely wrong with the benchmark setup. It seems that the benchmark has an environment factor with huge performance impact which is changing from run to run. I will try to setup the coreclr env on my machine and investigate what's going on here.

[AndyAyersMS]

A span-related test might not have been the best choice, since it probably requires netcoreapp2.0 or even later bits to run properly.

See what you find and in the meantime I'll look for something that doesn't require super-new builds.

[AndreyAkinshin]

I'll look for something that doesn't require super-new builds.

It would be great! If you find an example which can be run on netcoreapp1.1, I will appreciate it.

Meanwhile, I wrote a small R script which shows measured values for different MachineIds:

library(ggplot2)
df <- read.csv("TestSpanFillByteLength1000.csv")
df$Time <- strptime(df$Time, "%m/%d/%Y %I:%M %p")
p <- ggplot(df, aes(Time, Duration, colour = factor(MachineId)))
p + geom_point() + geom_line() + facet_wrap(~ MachineId)

Thus, it doesn't seem that there is a correlation between MachineId and measurements.

[AndyAyersMS]

Added another data set: https://gist.github.com/AndyAyersMS/e981f5540f1e1d7966279dc0c6ccfaf4#file-knucleotide_no_parallel-csv

This is an adaptation of one of the BenchmarksGames tests. Seems to flip quickly from one level to another. Should be fairly easy to get it running as netcoreapp1.1.

[adamsitnik]

I have created a new branch in BenchmarkDotNet repo: experiments and put single Span benchmark (SpanFillByte) there. Code

@AndreyAkinshin you need to install following things to get it running:
.NET Core 2.0 Preview 1
Visual Studio 2017 Preview 15.3 for Windows - it can be installed side by side with non-preview VS 2017

My sample results:

BenchmarkDotNet=v0.10.6.20170524-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742192 Hz, Resolution=364.6718 ns, Timer=TSC
dotnet cli version=2.0.0-preview1-005977
  [Host]     : .NET Core 4.6.25302.01, 64bit RyuJIT
  DefaultJob : .NET Core 4.6.25302.01, 64bit RyuJIT

Method	Length	Mean	Error	StdDev
TestSpanFillByte	10	3.871 ns	0.0753 ns	0.0704 ns
TestSpanFillByte	100	5.030 ns	0.0673 ns	0.0629 ns
TestSpanFillByte	1000	28.003 ns	0.1068 ns	0.0947 ns

@AndyAyersMS We have supported .NET Core 2.0 for a long time now, the Kestrel team is using BenchmarkDotNet extensively ;) If you would like to use it you can just install the latest version of BenchmarkDotNet package.

Andrey did not mention it but in order to have more accurate results and avoid some side effects (like GC self tunning) we always generate, build and execute separate process for every benchmark. We support C#, F#, VB, .NET 4.6+, .NET Core 1.1+ and Mono. We also have UAP on one of our branches which is almost finished.
The downside of this is that it takes more time to run the benchmarks and currently we don't support local core clr builds. But we also support custom settings like NetCoreAppImplicitPackageVersion or RuntimeFrameworkVersion so if you set the runtime version in explicit way and it's installed on your box you can run it for different runtime versions.

Please let me know if you need any help.