Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an optional way to configure storage to remember the PerfectInvocationCount, IterationCount and UnrollFactor #704

Closed
adamsitnik opened this issue Mar 26, 2018 · 2 comments

Comments

@adamsitnik
Copy link
Member

Scenario: I want to run a lot (thousands) of benchmarks for .NET framework itself. The first step that BDN does in pilot stage is finding the perfect invocation count.

My goal: add an option to remember this value (+ iteration count + unroll factor) to do it only once for new benchmarks and then read it from some storage.

My initial idea is following:

  1. add new interface IStorage to BenchmarkDotNet.dll
  2. add a nullable property to the IConfig which allows configuring IStorage
  3. Implement two classes that implement the interface. One for some lightweight file-db (like LiteDb) and one for some popular and cheap cloud db (like AzureTableStorage). Add it to new packages BenchmarkDotNet.Storage.$NameOfTheDb

@AndreyAkinshin what do you think about this idea? It would another step to improve the perf (#550)

@adamsitnik adamsitnik added this to the v0.11.0 milestone Mar 26, 2018
@adamsitnik adamsitnik self-assigned this Mar 26, 2018
@AndreyAkinshin
Copy link
Member

@adamsitnik, sorry for the late reply. Here are some thoughts:

  1. We can't completely skip the pilot change. E.g., if you make a 100x perf improvement or just run a benchmark on a very slow computer, we should calibrate the invocation count. However, it's a pretty good idea to automatically configure the initial invocation count. In simple cases, the pilot stage should be finished after the first iteration.
  2. We should keep the benchmark source code, config, and predefined values together (in the repository). If you have any settings in an external storage, it means that your benchmarks are not completely reproducible. Also, you can't test such benchmarks offline.
  3. A plain text format is preferred. Each predefined value contain 3 values: benchmark FQN, property name (e.g. InitialInvocationCount), property value. Even if you have thousands of benchmark, we can instantly load a file with thousands of lines (e.g. one line per benchmark) and parse them. I think that we don't need any db here (unless you are going to run billions of benchmarks). Also, it's much simpler to debug and experiment with plain text files.
  4. We can introduce an additional boolean property in the config which automatically generates such file.

@adamsitnik adamsitnik modified the milestones: v0.11.0, v0.11.x May 27, 2018
@adamsitnik
Copy link
Member Author

Closing as I don't need it anymore

@adamsitnik adamsitnik reopened this Oct 5, 2022
@adamsitnik adamsitnik closed this as not planned Won't fix, can't repro, duplicate, stale Oct 5, 2022
@adamsitnik adamsitnik removed this from the v0.13.x milestone Oct 5, 2022
@adamsitnik adamsitnik removed their assignment Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants