Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enables benchmarking betweeen different Nuget packages #922

Merged
merged 14 commits into from Oct 25, 2018

Conversation

Projects
None yet
3 participants
@Shazwazza
Copy link
Contributor

commented Oct 22, 2018

referenced issue: #290

Some notes on this PR:

  • Original PR was here #805 but the code became out-dated with recent changes to the benchmark.net project. There are some interesting notes about running benchmarks between Nuget versions #805 (comment) (will summarize more below)
  • This only supports running with the CsProjCoreToolchain, I've added a couple TODO notes in the Roslyn/Generator.cs since I think it might be possible to use the roslyn toolchain for this but I think will require a bunch of manual nuget.exe work (?)
  • Have added an IntroNuget sample. As per the notes mentioned above, this uses Reflection in the sample so that we don't need a direct reference to the Newtonsoft.Json, however if you think it would be more clear as far as sample code goes, I can add a direct reference to the minimum version tested of Newtsonsoft.Json to demonstrate how a benchmark project should be setup to test between different Nuget versions... just let me know.

How to benchmark between Nuget versions

Ideally this is how it is done:

  • The benchmark project references the minimum version of the nuget dependency that is being tested in the list of Jobs so that reflection doesn't have to be used to call the APIs and it builds
  • The list of Jobs specifies the exact version that will be tested and that is what will be resolved from Nuget
  • So long as the API signatures haven't changed between all of these versions then it should 'just work'

When each benchmark is run, it will add the correct nuget package dependency and build the project with that specific version and run the benchmarks against it.

This scenario will not work if API signatures have changed between versions. In those cases, different benchmark projects would need to be setup for the same API surface area and then the resulting reports could be combined.

Shazwazza added some commits Jun 25, 2018

Merge remote-tracking branch 'source/master' into 290-nuget-support
# Conflicts:
#	src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliBuilder.cs
#	src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommandExecutor.cs
#	src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliGenerator.cs
#	src/BenchmarkDotNet/Toolchains/Roslyn/Generator.cs
@adamsitnik
Copy link
Member

left a comment

@Shazwazza great PR! Honestly I was afraid that this feature will require a lot of code changes but you made great reuse of existing components and wrote it in really simple and smart way! 👍

Before we merge it could you please:

  1. Simplify the IntroNuget and add a simple doc page to our docs? You can take a look into docs\articles\samples for examples.
  2. Add at least one test? Could be sth like:
public class NugetReferenceTests : BenchmarkTestExecutor
{
    public NugetReferenceTests(ITestOutputHelper output) : base(output) { }

    [Fact]
    public void UserCanSpecifyCustomNuGetPackageDependency()
    {
        var toolchain = RuntimeInformation.IsFullFramework
            ? CsProjClassicNetToolchain.Current.Value // this .NET toolchain will do the right thing, the default RoslynToolchain does not support it
            : CsProjCoreToolchain.Current.Value;

        var job = Job.Dry.With(toolchain).WithNuget("Newtonsoft.Json", "11.0.2");
        var config = CreateSimpleConfig(job: job);

        CanExecute<WithCallToNewtonsoft>(config);
    }

    public class WithCallToNewtonsoft
    {
        [Benchmark] public void CallNewton() { } // the code that you added to IntroNuget would be great
    }
}
Show resolved Hide resolved src/BenchmarkDotNet/Jobs/JobExtensions.cs Outdated
Show resolved Hide resolved src/BenchmarkDotNet/Jobs/NugetReference.cs Outdated
Show resolved Hide resolved src/BenchmarkDotNet/Toolchains/DotNetCli/DotNetCliCommand.cs Outdated
Show resolved Hide resolved src/BenchmarkDotNet/Toolchains/Roslyn/Generator.cs Outdated
Show resolved Hide resolved src/BenchmarkDotNet/Toolchains/Roslyn/Generator.cs Outdated
Show resolved Hide resolved samples/BenchmarkDotNet.Samples/IntroNuget.cs

Shazwazza added some commits Oct 25, 2018

Adds integration and unit test, ensures that multiple packages can be…
… added but not multiple of the same name, fixes version validation for pre-releases
Adds check to RoslynToolchain for nuget dependencies, adds unit tests…
… to show that the RoslynToolchain doesn't support Nuget refs
@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2018

@adamsitnik I've pushed a few more commits to address your review including a few tests, let me know if there's anything i've missed

@adamsitnik adamsitnik merged commit 92a7869 into dotnet:master Oct 25, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
license/cla All CLA requirements met.
Details
@adamsitnik

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

@Shazwazza thank you! great job!

@AndreyAkinshin AndreyAkinshin added this to the v0.11.2 milestone Oct 25, 2018

@Shazwazza

This comment has been minimized.

Copy link
Contributor Author

commented Oct 25, 2018

Whoohoo! awesome :) 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.