New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Cleanup and Setup between benchmarks #270

Closed
redknightlois opened this Issue Sep 30, 2016 · 4 comments

Comments

Projects
None yet
4 participants
@redknightlois
Contributor

redknightlois commented Sep 30, 2016

We have observed that there are attributes ([Setup] and [Cleanup]) that allow you to set before and after actions for a class of benchmarks. However, this is only performed before -and after- the entire set of benchmarks. As far I know this is an unsupported scenario (unless I am missing something)

Scenario: We want to benchmark insertions/deletions on a database.

To attempt to recreate real-life scenarios, we need to use a pre-filled database with plenty of data (like a few GBs of startup data). This could be done, in principle, by using the Setup attribute and copy the database on each run; however, afaik Setup/Cleanup only runs at the start/end of the entire bench.

Any ideas on how to accomplish that?

@russpowers

This comment has been minimized.

Show comment
Hide comment
@russpowers

russpowers Oct 21, 2016

Did you ever figure this out? I'm having the exact same problem benchmarking inserts into a tree that uses unsafe memory allocations. I need destroy the tree in between benchmark operations, but there doesn't seem to be a way to do that.

russpowers commented Oct 21, 2016

Did you ever figure this out? I'm having the exact same problem benchmarking inserts into a tree that uses unsafe memory allocations. I need destroy the tree in between benchmark operations, but there doesn't seem to be a way to do that.

@jbayardo

This comment has been minimized.

Show comment
Hide comment
@jbayardo

jbayardo Oct 21, 2016

The bug report is misleading. The way we actually found this works is:

  • Setup is run
  • Benchmark method is run
  • Cleanup is run

This happens for every benchmark method. What we did is, in our Setup we just have a check to see if the tree has been filled previously or not (this is easy to do in our case because it is persisted on to disk); since this is not measured, we avoid paying the cost in the benchmark itself. You can see sample code for this here: https://github.com/jbayardo/ravendb/blob/v4.0/bench/Voron.Benchmark/BTree/BTreeInsertRandom.cs .

In your case (if I understood correctly), there should be no trouble doing this, just use [Setup] and [Cleanup] as advertised and it should work correctly.

jbayardo commented Oct 21, 2016

The bug report is misleading. The way we actually found this works is:

  • Setup is run
  • Benchmark method is run
  • Cleanup is run

This happens for every benchmark method. What we did is, in our Setup we just have a check to see if the tree has been filled previously or not (this is easy to do in our case because it is persisted on to disk); since this is not measured, we avoid paying the cost in the benchmark itself. You can see sample code for this here: https://github.com/jbayardo/ravendb/blob/v4.0/bench/Voron.Benchmark/BTree/BTreeInsertRandom.cs .

In your case (if I understood correctly), there should be no trouble doing this, just use [Setup] and [Cleanup] as advertised and it should work correctly.

@russpowers

This comment has been minimized.

Show comment
Hide comment
@russpowers

russpowers Oct 22, 2016

Thanks for the reply. I'm still having the issue, even with setting OperationsPerInvoke to 1. On the first couple runs, things work as you say, but then it starts warming up and does more than one benchmark between setup and cleanup. According to their docs, Setup/Cleanup are run before/after each iteration, not each benchmark operation: https://perfdotnet.github.io/BenchmarkDotNet/HowItWorks.htm

I added some WriteLines to the code, and here's the output I get:

// *** Execute ***
// Launch: 1
CTOR
SETUP
Insert10MRandomUnique
CLEANUP

// Benchmark Process Environment Information:
// CLR=CORE, Arch=64-bit ? [RyuJIT]
// GC=Concurrent Workstation

SETUP
Insert10MRandomUnique
CLEANUP
SETUP
Insert10MRandomUnique
Insert10MRandomUnique
<duplicate key crash>

russpowers commented Oct 22, 2016

Thanks for the reply. I'm still having the issue, even with setting OperationsPerInvoke to 1. On the first couple runs, things work as you say, but then it starts warming up and does more than one benchmark between setup and cleanup. According to their docs, Setup/Cleanup are run before/after each iteration, not each benchmark operation: https://perfdotnet.github.io/BenchmarkDotNet/HowItWorks.htm

I added some WriteLines to the code, and here's the output I get:

// *** Execute ***
// Launch: 1
CTOR
SETUP
Insert10MRandomUnique
CLEANUP

// Benchmark Process Environment Information:
// CLR=CORE, Arch=64-bit ? [RyuJIT]
// GC=Concurrent Workstation

SETUP
Insert10MRandomUnique
CLEANUP
SETUP
Insert10MRandomUnique
Insert10MRandomUnique
<duplicate key crash>
@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 30, 2017

Member

Implemented, now we have [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup].

Member

AndreyAkinshin commented May 30, 2017

Implemented, now we have [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment