New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for run-once Setup and Clean-up with Parameters available #274

Closed
jbayardo opened this Issue Oct 3, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@jbayardo

jbayardo commented Oct 3, 2016

Scenario

Picture the following situation:

  • We want to benchmark the performance of a tree implementation for a database.
  • We are interested on measuring the behavior of the tree as its size grows really large.

Hence, in our [Setup] method, we create a huge tree (this is because we NEED the parameters to be set, which does not happen on the Constructor as far as I have tested). This is an expensive operation, taking a long time to finish, in part due to the number of insertions, but mostly due to the required I/O.

Problem

As of right now, every [Benchmark] we have is going to create a new tree, and the Cleanup method is going to remove it. Thus, every Benchmark takes a ton of time -and this is not due to the benchmark itself, but to the time required to recreate the conditions under which it should run-.

Next idea, load it up from disk whenever possible. Problem: the tree is not re-created when new parameters are set, unless we explicitly check for that. This can not be solved by doing things in the class's constructor, because the constructor does not have the parameters set yet, so this must be done every time we Setup a Bechmark.

Proposed Flow

It would be much easier if there was an option to run a "Pre-Setup" that would only run when parameters change. The way I picture this is:

  1. Class is instantiated
  2. Parameters are loaded
  3. Pre-Run Setup is called
  4. For each Benchmark iteration: Run setup, Run benchmark, Run Cleanup
  5. Post-Cleanup is called
  6. Instance is destroyed

I'm not sure how this fits in with BenchmarkDotNet's current way of doing things. Is there anything that allows for a similar workflow?.

Thanks!

@jbayardo jbayardo changed the title from Support for specific Setup and Cleanup tasks to Support for run-once Setup and Clean-up with Parameters available Oct 3, 2016

@redknightlois

This comment has been minimized.

Show comment
Hide comment
@redknightlois

redknightlois Oct 3, 2016

Contributor

Actually, we arrived at the same realization from different points of view. I was at the gym when I figured it out, and I was going to file the feature request after arriving to the office :P.

The solution for this is quite simple. The only thing that we need is a benchmark timestamp that gets baked in on the source-code generation process.

Lets say:

public class Benchmark
{
        [BenchmarkTimestamp]
        public void Timestamp
        {
              get { return ConstantGeneratedOnCodeGeneration };
        }
}

For the most part (on our particular problem) we can just go and build the database state with the timestamp and the next setups are going to know it exist and copy the file to a new location and work from there.

Is it viable?

Contributor

redknightlois commented Oct 3, 2016

Actually, we arrived at the same realization from different points of view. I was at the gym when I figured it out, and I was going to file the feature request after arriving to the office :P.

The solution for this is quite simple. The only thing that we need is a benchmark timestamp that gets baked in on the source-code generation process.

Lets say:

public class Benchmark
{
        [BenchmarkTimestamp]
        public void Timestamp
        {
              get { return ConstantGeneratedOnCodeGeneration };
        }
}

For the most part (on our particular problem) we can just go and build the database state with the timestamp and the next setups are going to know it exist and copy the file to a new location and work from there.

Is it viable?

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 30, 2017

Member

Implemented, now we have [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup].

Member

AndreyAkinshin commented May 30, 2017

Implemented, now we have [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment