New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSBuild parameters are not passed to generated benchmark project #466

Closed
xplicit opened this Issue Jun 10, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@xplicit

xplicit commented Jun 10, 2017

I run the following command to build and run benchmarks:

dotnet restore /p:BASELINE=1
dotnet build /p:BASELINE=1 -c Release
dotnet exec -c Release -f netcoreapp1.1 bin/Release/netcoreapp1.1/benchmarks.dll

But BenchmarkDotNet creates its own wrapping project and does not pass /p:BASELINE=1 to msbuild. Here is content of BDN.Generated.sh

call dotnet restore --no-dependencies
call dotnet build --framework netcoreapp1.1 --configuration Release

while it should be

call dotnet restore /p:BASELINE=1 --no-dependencies
call dotnet build /p:BASELINE=1 --framework netcoreapp1.1 --configuration Release
@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Jun 10, 2017

Member

Hello @xplicit !

Thanks for a very well described problem!

We have a feature that allows using parameters in our benchmarks. It's called params, here you can find full description.

Could you explain me your motivation behind passing custom parameters as console app parameters to the generated process? Or maybe our [Params] feature can be enough for you?

Member

adamsitnik commented Jun 10, 2017

Hello @xplicit !

Thanks for a very well described problem!

We have a feature that allows using parameters in our benchmarks. It's called params, here you can find full description.

Could you explain me your motivation behind passing custom parameters as console app parameters to the generated process? Or maybe our [Params] feature can be enough for you?

@xplicit

This comment has been minimized.

Show comment
Hide comment
@xplicit

xplicit Jun 10, 2017

The idea of passing msbuild params is to make benchmarks for different assembly versions of the same library. When I pass /p:BASELINE=1 I compile and run benchmarks with baseline version of lib, when I did not pass it I compile and run benchmarks with current code of library.

So the high-level idea is to run v1 of mylib.dll and v2 of mylib.dll and see if there are performance regressions during the code changes.

xplicit commented Jun 10, 2017

The idea of passing msbuild params is to make benchmarks for different assembly versions of the same library. When I pass /p:BASELINE=1 I compile and run benchmarks with baseline version of lib, when I did not pass it I compile and run benchmarks with current code of library.

So the high-level idea is to run v1 of mylib.dll and v2 of mylib.dll and see if there are performance regressions during the code changes.

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Jun 11, 2017

Member

In the future, I want to implement a feature which allows setting custom environment variable in each job, it will help to support features like this: #262
@adamsitnik, maybe it makes sense to introduce also an ability to setting build arguments?

Member

AndreyAkinshin commented Jun 11, 2017

In the future, I want to implement a feature which allows setting custom environment variable in each job, it will help to support features like this: #262
@adamsitnik, maybe it makes sense to introduce also an ability to setting build arguments?

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Jun 16, 2017

Member

@xplicit the idea sounds very cool! But I can't find any information about this parameter. Could you provide me some link? I would like to implement and test it.

Member

adamsitnik commented Jun 16, 2017

@xplicit the idea sounds very cool! But I can't find any information about this parameter. Could you provide me some link? I would like to implement and test it.

@xplicit

This comment has been minimized.

Show comment
Hide comment
@xplicit

xplicit Jun 18, 2017

@adamsitnik BASELINE is custom property which I define from msbuild command line using /p parameter (see https://msdn.microsoft.com/ru-ru/library/ms164311.aspx /property parameter). Then I use it in conditional csproj. Something like that:

  <Choose>
    <When Condition=" '$(BASELINE)' == '' ">
      <ItemGroup>
        <ProjectReference Include="..\..\src\ServiceStack.Text\ServiceStack.Text.csproj" Version="1.0.*" />
      </ItemGroup>
    </When>
    <When Condition=" '$(BASELINE)' != '' ">
      <ItemGroup Condition=" '$(TargetFramework)' == 'net46' ">
        <PackageReference Include="ServiceStack.Text" Version="4.5.8" />
      </ItemGroup>
      <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
        <PackageReference Include="ServiceStack.Text.Core" Version="1.0.38" />
      </ItemGroup>
    </When>
  </Choose>

So when I pass /p:BASELINE=1 into msbuild I link with "baseline" version of library, when I do not pass it I link with current developing project.

xplicit commented Jun 18, 2017

@adamsitnik BASELINE is custom property which I define from msbuild command line using /p parameter (see https://msdn.microsoft.com/ru-ru/library/ms164311.aspx /property parameter). Then I use it in conditional csproj. Something like that:

  <Choose>
    <When Condition=" '$(BASELINE)' == '' ">
      <ItemGroup>
        <ProjectReference Include="..\..\src\ServiceStack.Text\ServiceStack.Text.csproj" Version="1.0.*" />
      </ItemGroup>
    </When>
    <When Condition=" '$(BASELINE)' != '' ">
      <ItemGroup Condition=" '$(TargetFramework)' == 'net46' ">
        <PackageReference Include="ServiceStack.Text" Version="4.5.8" />
      </ItemGroup>
      <ItemGroup Condition=" '$(TargetFramework)' == 'netcoreapp1.1' ">
        <PackageReference Include="ServiceStack.Text.Core" Version="1.0.38" />
      </ItemGroup>
    </When>
  </Choose>

So when I pass /p:BASELINE=1 into msbuild I link with "baseline" version of library, when I do not pass it I link with current developing project.

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Jun 18, 2017

Member

@xplicit now I get it, thanks! I will try to implement it this week

my long term plan is to add similar feature to BenchmarkDotNet. It could be sth like this:

Job.Default.WithReferences(new PackageReference("PackageName", "1.0.0"));
Job.Default.WithReferences(new PackageReference("PackageName", "2.0.0"));
Job.Default.WithReferences(new ProjectReference("pathToCsProj"));

I wanted to implement it a long time ago, but then project.json was being abandoned and I wanted to wait.

Member

adamsitnik commented Jun 18, 2017

@xplicit now I get it, thanks! I will try to implement it this week

my long term plan is to add similar feature to BenchmarkDotNet. It could be sth like this:

Job.Default.WithReferences(new PackageReference("PackageName", "1.0.0"));
Job.Default.WithReferences(new PackageReference("PackageName", "2.0.0"));
Job.Default.WithReferences(new ProjectReference("pathToCsProj"));

I wanted to implement it a long time ago, but then project.json was being abandoned and I wanted to wait.

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 3, 2017

Member

@xplicit sorry for the delay, here it is:

It's possible now, with new thing called Arguments. MsBuild argument does exactly what you need. Sample config:

public class ConfigWithCustomArguments : ManualConfig
{
    public ConfigWithCustomArguments()
    {
        Add(Job.Core.With(new[] { new MsBuildArgument("/p:BASELINE=1") }).WithId("Baseline=1"));
        Add(Job.Core.WithId("No baseline"));
    }
}

[Config(typeof(ConfigWithCustomArguments))]
public class Sample
{
    [Benchmark]
    public void Benchmark() { }
}

With such config the Benchmark is going to be executed for both jobs.

As soon as this build finishes, you should be able to grab the latest NuGet package from our CI feed https://ci.appveyor.com/nuget/benchmarkdotnet

I am closing it right now, please feel free to reopen if something does not work as expected

Member

adamsitnik commented Sep 3, 2017

@xplicit sorry for the delay, here it is:

It's possible now, with new thing called Arguments. MsBuild argument does exactly what you need. Sample config:

public class ConfigWithCustomArguments : ManualConfig
{
    public ConfigWithCustomArguments()
    {
        Add(Job.Core.With(new[] { new MsBuildArgument("/p:BASELINE=1") }).WithId("Baseline=1"));
        Add(Job.Core.WithId("No baseline"));
    }
}

[Config(typeof(ConfigWithCustomArguments))]
public class Sample
{
    [Benchmark]
    public void Benchmark() { }
}

With such config the Benchmark is going to be executed for both jobs.

As soon as this build finishes, you should be able to grab the latest NuGet package from our CI feed https://ci.appveyor.com/nuget/benchmarkdotnet

I am closing it right now, please feel free to reopen if something does not work as expected

@adamsitnik adamsitnik closed this Sep 3, 2017

@bluetentacle

This comment has been minimized.

Show comment
Hide comment
@bluetentacle

bluetentacle Aug 24, 2018

I'm trying to benchmark the difference between the current and past versions of a library, using the MSBuildArgument command.

I couldn't make it work.

Here's my config:

Add(Job.Core.With(new[] { new MsBuildArgument("/p:PREVIOUS_VERSION=1.1.5")}).WithId("Before").AsBaseline());
Add(Job.Core.WithId("Now"));

And here's my benchmark csproj:

  <ItemGroup >
    <ProjectReference Include="..\..\src\MyAssembly.Hosting\MyAssembly.csproj" Condition=" '$(PREVIOUS_VERSION)' == '' " />
    <PackageReference Include="MyAssembly.Hosting" Version="$(PREVIOUS_VERSION)" Condition=" '$(PREVIOUS_VERSION)' != '' " />
  </ItemGroup>

It builds but at runtime it craps out. The "Before" job complains that it can't find the version 1.1.6 of MyAssembly, even though the PREVIOUS_VERSION MSBuild parameter instructs it to use version 1.1.5.

It seems that this is because the benchmark project (the one I define the benchmarks) is only built once--without the MSBuild parameter. Indeed, when I look at the temporary folders that BenchmarkDotNet generates, all of the benchmark assemblies are the same.

Any suggestions on how I can actually make my comparison benchmark work?

bluetentacle commented Aug 24, 2018

I'm trying to benchmark the difference between the current and past versions of a library, using the MSBuildArgument command.

I couldn't make it work.

Here's my config:

Add(Job.Core.With(new[] { new MsBuildArgument("/p:PREVIOUS_VERSION=1.1.5")}).WithId("Before").AsBaseline());
Add(Job.Core.WithId("Now"));

And here's my benchmark csproj:

  <ItemGroup >
    <ProjectReference Include="..\..\src\MyAssembly.Hosting\MyAssembly.csproj" Condition=" '$(PREVIOUS_VERSION)' == '' " />
    <PackageReference Include="MyAssembly.Hosting" Version="$(PREVIOUS_VERSION)" Condition=" '$(PREVIOUS_VERSION)' != '' " />
  </ItemGroup>

It builds but at runtime it craps out. The "Before" job complains that it can't find the version 1.1.6 of MyAssembly, even though the PREVIOUS_VERSION MSBuild parameter instructs it to use version 1.1.5.

It seems that this is because the benchmark project (the one I define the benchmarks) is only built once--without the MSBuild parameter. Indeed, when I look at the temporary folders that BenchmarkDotNet generates, all of the benchmark assemblies are the same.

Any suggestions on how I can actually make my comparison benchmark work?

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Aug 25, 2018

Member

Hi @bluetentacle

I am afraid that you need to wait until we make some progress with #805

Member

adamsitnik commented Aug 25, 2018

Hi @bluetentacle

I am afraid that you need to wait until we make some progress with #805

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment