Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Add support for Cleanup and Setup between benchmarks #270
We have observed that there are attributes ([Setup] and [Cleanup]) that allow you to set before and after actions for a class of benchmarks. However, this is only performed before -and after- the entire set of benchmarks. As far I know this is an unsupported scenario (unless I am missing something)
Scenario: We want to benchmark insertions/deletions on a database.
To attempt to recreate real-life scenarios, we need to use a pre-filled database with plenty of data (like a few GBs of startup data). This could be done, in principle, by using the Setup attribute and copy the database on each run; however, afaik Setup/Cleanup only runs at the start/end of the entire bench.
Any ideas on how to accomplish that?
The bug report is misleading. The way we actually found this works is:
This happens for every benchmark method. What we did is, in our Setup we just have a check to see if the tree has been filled previously or not (this is easy to do in our case because it is persisted on to disk); since this is not measured, we avoid paying the cost in the benchmark itself. You can see sample code for this here: https://github.com/jbayardo/ravendb/blob/v4.0/bench/Voron.Benchmark/BTree/BTreeInsertRandom.cs .
In your case (if I understood correctly), there should be no trouble doing this, just use [Setup] and [Cleanup] as advertised and it should work correctly.
Thanks for the reply. I'm still having the issue, even with setting OperationsPerInvoke to 1. On the first couple runs, things work as you say, but then it starts warming up and does more than one benchmark between setup and cleanup. According to their docs, Setup/Cleanup are run before/after each iteration, not each benchmark operation: https://perfdotnet.github.io/BenchmarkDotNet/HowItWorks.htm
I added some WriteLines to the code, and here's the output I get: