New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange behaviour when benchmark project is build in debug mode #561

Closed
slavam2605 opened this Issue Sep 28, 2017 · 6 comments

Comments

Projects
None yet
3 participants
@slavam2605

slavam2605 commented Sep 28, 2017

I create a project and make a benchmark, then run it. After that I change the benchmark and run it again but I got results for old benchmark.
It is .NET Core 2.0 project and it was always build in debug mode.

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Sep 28, 2017

Member

It seems that the problem is: BenchmarkDotNet builds .NET Core project in Release so it uses the Release build of the original project. However, if we are making any changes of the original project in Debug, the generated benchmark will still use obsolete Release assemblies.

Member

AndreyAkinshin commented Sep 28, 2017

It seems that the problem is: BenchmarkDotNet builds .NET Core project in Release so it uses the Release build of the original project. However, if we are making any changes of the original project in Debug, the generated benchmark will still use obsolete Release assemblies.

@adamsitnik adamsitnik self-assigned this Oct 15, 2017

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Oct 16, 2017

Member

@slavam2605 thanks for reporting this bug!

@AndreyAkinshin I have found the reason for this.

To optimize the time of dotnet build I have used --no-dependencies config switch. It builds only given project, without dependencies. It gives us huge perf gain, especially if somebody is using [Params], which would create a lot of projects and here we can avoid of the rebuild of all the dependencies.

The problem is when some user:

  1. Builds the entire solution in Release
  2. Makes a change
  3. Runs benchmarks in Debug (if you don't specify configuration the Debug is the default one)
  4. We don't rebuild all the dependencies, we don't get error because the files (old) are there and we run them.

Currently I have two ideas:

  1. We can drop the optimization, but it will prolong all builds (I don't like this idea)
  2. We can make the JitOptimizationsValidator.FailOnError a mandatory validator. In that case if user runs dotnet run without -c Release he\she gets an critical error. If the user runs dotnet run -c Release then dotnet cli rebuilds everything for us only once, and afterwards we still use --no-dependencies, we have good perf and correctness ;)

@AndreyAkinshin what do you think? I would like to change the default behavior to fail for Debug ;)

Member

adamsitnik commented Oct 16, 2017

@slavam2605 thanks for reporting this bug!

@AndreyAkinshin I have found the reason for this.

To optimize the time of dotnet build I have used --no-dependencies config switch. It builds only given project, without dependencies. It gives us huge perf gain, especially if somebody is using [Params], which would create a lot of projects and here we can avoid of the rebuild of all the dependencies.

The problem is when some user:

  1. Builds the entire solution in Release
  2. Makes a change
  3. Runs benchmarks in Debug (if you don't specify configuration the Debug is the default one)
  4. We don't rebuild all the dependencies, we don't get error because the files (old) are there and we run them.

Currently I have two ideas:

  1. We can drop the optimization, but it will prolong all builds (I don't like this idea)
  2. We can make the JitOptimizationsValidator.FailOnError a mandatory validator. In that case if user runs dotnet run without -c Release he\she gets an critical error. If the user runs dotnet run -c Release then dotnet cli rebuilds everything for us only once, and afterwards we still use --no-dependencies, we have good perf and correctness ;)

@AndreyAkinshin what do you think? I would like to change the default behavior to fail for Debug ;)

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Oct 16, 2017

Member
  1. I also don't like this idea.
  2. LGTM, let's do it. However, it makes sense to add a few lines in the documentation about how to disable the mandatory check (for people who want to perform debugging instead of benchmarking).
Member

AndreyAkinshin commented Oct 16, 2017

  1. I also don't like this idea.
  2. LGTM, let's do it. However, it makes sense to add a few lines in the documentation about how to disable the mandatory check (for people who want to perform debugging instead of benchmarking).
@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Oct 16, 2017

Member

@AndreyAkinshin do you think that we should allow our users to do the debugging?

I was thinking about leaving the debugging only for us, by using the ugly #if DEBUG conditional switch.

I am asking because if I make it a mandatory validator, then today there is no way to disable it. If I make it part of DefaultConfig, then there is a way to omit it, but the users with ManualConfigs might still get into this problem.

Member

adamsitnik commented Oct 16, 2017

@AndreyAkinshin do you think that we should allow our users to do the debugging?

I was thinking about leaving the debugging only for us, by using the ugly #if DEBUG conditional switch.

I am asking because if I make it a mandatory validator, then today there is no way to disable it. If I make it part of DefaultConfig, then there is a way to omit it, but the users with ManualConfigs might still get into this problem.

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Oct 16, 2017

Member

@adamsitnik, we are also our users. =) #if DEBUG works great when you work with the source code of BenchmarkDotNet. However, sometimes I have to debug the BenchmarkDotNet package from NuGet (Release).
Also sometimes I need it for comparing Debug and Release builds (it's useful in my research which is related to Roslyn/RyuJIT optimizations).

Member

AndreyAkinshin commented Oct 16, 2017

@adamsitnik, we are also our users. =) #if DEBUG works great when you work with the source code of BenchmarkDotNet. However, sometimes I have to debug the BenchmarkDotNet package from NuGet (Release).
Also sometimes I need it for comparing Debug and Release builds (it's useful in my research which is related to Roslyn/RyuJIT optimizations).

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Oct 17, 2017

Member

@AndreyAkinshin I think I found the solution.

The default (for default config) is the one with errors.
The mandatory validator is the one with warning so everybody, even power users get warning.

Member

adamsitnik commented Oct 17, 2017

@AndreyAkinshin I think I found the solution.

The default (for default config) is the one with errors.
The mandatory validator is the one with warning so everybody, even power users get warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment