Same NuGet version used when benchmarking different packages

[blairconrad]

As discussed in Gitter, my attempt to benchmark two different NuGet versions of FakeItEasy seem to be running the benchmarks against a single version. I'm using BenchmarkDotNet version 0.11.1.821 from the nightlies, and started from the example configuration added in #922.

I quote the Gitter conversation:

Hmmm. As I continue working with this, I think it's doing the wrong thing. I don't know if I'm doing something wrong or if there's a bug. I think we're always running the code from one NuGet version. I have a benchmark that's dramatically faster with my 4.10.0… version than the 4.9.1, but if I run the two jobs, I get numbers in line with 4.10 only:

BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
  [Host] : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  4.9.1  : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT

Job=4.9.1  NugetReferences=FakeItEasy 4.9.1  Toolchain=.NET Core 2.1  
IterationCount=1  LaunchCount=1  WarmupCount=10  

Method	Mean	Error
Literal	135.1 us	NA

---

BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
  [Host]                      : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  4.10.0-make-servicelocat001 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT

Job=4.10.0-make-servicelocat001  NugetReferences=FakeItEasy 4.10.0-make-servicelocat001  Toolchain=.NET Core 2.1  
IterationCount=1  LaunchCount=1  WarmupCount=10  

Method	Mean	Error
Literal	15.23 us	NA

---

BenchmarkDotNet=v0.11.1.821-nightly, OS=Windows 10.0.17134.345 (1803/April2018Update/Redstone4)
Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge), 1 CPU, 8 logical and 4 physical cores
Frequency=2143568 Hz, Resolution=466.5119 ns, Timer=TSC
.NET Core SDK=2.1.402
  [Host]                      : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  4.10.0-make-servicelocat001 : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT
  4.9.1                       : .NET Core 2.1.4 (CoreCLR 4.6.26814.03, CoreFX 4.6.26814.02), 64bit RyuJIT

Toolchain=.NET Core 2.1  IterationCount=1  LaunchCount=1  
WarmupCount=10  

Method	Job	NugetReferences	Mean	Error
Literal	4.10.0-make-servicelocat001	FakeItEasy 4.10.0-make-servicelocat001	15.24 us	NA
Literal	4.9.1	FakeItEasy 4.9.1	15.31 us	NA

My benchmark project references 4.9.1, and build my configuration looks like this:

public Config(params string[] versions)
{
    // Specify jobs with different versions of the same Nuget package to benchmark.
    // The Nuget versions referenced on these jobs must be greater or equal to the 
    // same Nuget version referenced in this benchmark project.
    foreach (var version in versions)
    {
        Add(Job.MediumRun
            .WithIterationCount(1)
            .WithLaunchCount(1)
            .With(CsProjCoreToolchain.Current.Value)
            .WithNuget("FakeItEasy", version)
            .WithId(version));
    }
    Add(DefaultConfig.Instance.GetColumnProviders().ToArray());
    Add(DefaultConfig.Instance.GetLoggers().ToArray());
    Add(CsvExporter.Default);
    Add(MarkdownExporter.GitHub);
}

the 4.10.0-make-servicelocat001 package is not an official one, but existed in a local NuGet source I had on my computer.

FakeItEasy.Benchmark.zip contains the source I used to reproduce, including the output I got when I ran benchmark.ps1, which attempts to benchmark 4.9.1 alone, 4.10.0-make-servicelocat001 alone, and the two together. The last attempt gives the suspect results.
I included nuget/4.10.0-make-servicelocat001.nupkg in case someone wants to reproduce.

In the .log file for the combined run, I see start dotnet add package FakeItEasy -v 4.10.0-make-servicelocat001 only. No similar line for 4.9.1, even when I change my .csproj to reference something older, like 4.8.0.

I'll try to debug myself, but am occupied for the rest of today. With any luck, I'm doing something silly.

[AndreyAkinshin]

@Shazwazza could you take a look?

[blairconrad]

Working theory:

BenchmarkDotNet.Running.BenchmarkPartitioner.BenchmarkRuntimePropertiesComparer.Equals could use an additional

if (AreDifferent(jobX.Infrastructure.NugetReferences, jobY.Infrastructure.NugetReferences))
    return false;

Trying to build and test now.

[blairconrad]

I think we have a winner.

Method	Job	NugetReferences	Mean	Error
Literal	4.10.0-make-servicelocat001	FakeItEasy 4.10.0-make-servicelocat001	15.66 us	NA
Literal	4.9.1	FakeItEasy 4.9.1	88.65 us	NA

I'll try to figure out a test and whip up a PR, if you'd like.

[Shazwazza]

Yup I can have a look tomorrow if you haven't already beat me to it

[blairconrad]

@Shazwazza, I'm making good progress on a PR. If it suits you, save your energy for reviewing it. 😁