initial release notes (#833)

dotnet · Jul 19, 2018 · a7ecdae · a7ecdae
1 parent a298c2e
commit a7ecdae
Showing 1 changed file with 305 additions and 0 deletions.
diff --git a/docs/_changelog/header/v0.11.0.md b/docs/_changelog/header/v0.11.0.md
@@ -0,0 +1,305 @@
+This is one of the biggest releases of BenchmarkDotNet ever.
+
+A big part of the features and bug fixes were implemented to meet the enterprise requirements of Microsoft to make it possible to port CoreCLR, CoreFX, and CoreFXLab to BenchmarkDotNet.
+
+The release would not be possible without many contributions from amazing community members. This release is a combined effort. We build BenchmarkDotNet together to make benchmarking .NET code easy and available to everyone for free!
+
+# New docs!
+
+TODO: Andrey could you please fill in this part?
+
+# Performance improvements
+
+BenchmarkDotNet needs to be capable of running few thousands of CoreFX and CoreCLR benchmarks in an acceptable amount of time. The code itself was already optimized so we needed architectural and design changes to meet this requirement.
+
+## Generate one executable per runtime settings
+
+To ensure that the side effects of one benchmark run does not affect another benchmark run BenchmarkDotNet generates, builds and runs every benchmark in a dedicated process. So far we were generating and building one executable per benchmark, now we generate and build one executable per runtime settings. So if you want to run ten thousands of benchmarks for .NET Core 2.1 we are going to generate and build single executable, not ten thousand. If you target multiple runtimes the build is going to be executed in parallel. Moreover, if one of the parallel builds fail it's going to be repeated in a sequential way.
+
+Previously the time to generate and build 650 benchmarks from our Samples project was **one hour**. Now it's something around **13 seconds** which means **276 X** improvement for this particular scenario. You can see the changes [here](https://github.com/dotnet/BenchmarkDotNet/issues/699).
+
+## Don't execute long operations more than once per iteration
+
+BenchmarkDotNet was designed to allow for very accurate and stable micro-benchmarking. One of the techniques that we use is manual loop unrolling. In practice, it meant that for every iteration we were executing the benchmark at least 16 times (the default `UnrollFactor` value). It was of course not desired for the very time-consuming benchmarks.
+
+So far this feature was always enabled by default and users would need to configure `UnrollFactor=1` to disable it. Now BenchmarkDotNet is going to discover such scenario and don't perform manual loop unrolling for the very time-consuming benchmarks. BenchmarkDotNet uses `Job.IterationTime` setting (the default is 0.5s) in the Pilot Experiment stage to determine how many times given benchmark should be executed per iteration.
+
+Example:
+
+```cs
+public class Program
+{
+    static void Main() => BenchmarkRunner.Run<Program>();
+
+    [Benchmark]
+    public void Sleep1s() => Thread.Sleep(TimeSpan.FromSeconds(1));
+}
+```
+
+Time to run with the previous version: **374 seconds**. With `0.11.0` it's **27 seconds** which gives us almost **14 X** improvement. A good example of benchmarks that are going to benefit from this change are computer game benchmarks and ML.NET benchmarks. You can see the changes [here](https://github.com/dotnet/BenchmarkDotNet/pull/760) and [here](https://github.com/dotnet/BenchmarkDotNet/pull/771).
+
+## Exposing more configuration settings
+
+The default settings were configured to work well with every scenario. Before running the benchmark, BenchmarkDotNet does not know anything about it. This is why it performs many warmup iterations before running the benchmarks.
+
+When you author benchmarks and run them many times you can come up with custom settings that produce similar results but in a shorter manner of time. To allow you to do that we have exposed:
+
+* `Job.MinIterationCount` (default value is 15)
+* `Job.MaxIterationCount` (default value is 100)
+* `Job.MinWarmupIterationCount` (default value is 6)
+* `Job.MaxWarmupIterationCount` (default value is 50)
+
+# User Experience
+
+One of the biggest success factors of BenchmarkDotNet is a great user experience. The tool just works as expected and makes your life easy. We want to make it even better!
+
+## .NET Standard 2.0
+
+We have ported BenchmarkDotNet to .NET Standard 2.0 and thanks to that we were able to not only simplify our code and build process but also merge `BenchmarkDotNet.Core.dll` and `BenchmarkDotNet.Toolchains.Roslyn.dll` into `BenchmarkDotNet.dll`. We still support .NET 4.6 but we have dropped .NET Core 1.1 support. More information and full discussion can be found [here](https://github.com/dotnet/BenchmarkDotNet/pull/688).
+
+**Note:** Our `BenchmarkDotNet.Diagnostics.Windows` package which uses `EventTrace` to implement ETW-based diagnosers was also ported to .NET Standard 2.0 and you can now use all the ETW diagnosers with .NET Core on Windows. We plan to add EventPipe support and make this page fully cross-platform and Unix compatible soon.
+
+## Using complex types as benchmark arguments
+
+So far we have required the users to implement `IParam` interface to make the custom complex types work as benchmark arguments/parameters. This has changed, now the users can use any complex types as arguments and it will just work ([more](https://github.com/dotnet/BenchmarkDotNet/pull/754)).
+
+```cs
+public class Program
+{
+    static void Main(string[] args) => BenchmarkRunner.Run<Program>();
+
+    public IEnumerable<object> Arguments()
+    {
+        yield return new Point2D(10, 200);
+    }
+
+    [Benchmark]
+    [ArgumentsSource(nameof(Arguments))]
+    public int WithArgument(Point2D point) => point.X + point.Y;
+}
+
+public class Point2D
+{
+    public int X, Y;
+
+    public Point2D(int x, int y)
+    {
+        X = x;
+        Y = y;
+    }
+
+    public override string ToString() => $"[{X},{Y}]";
+}
+```
+
+**Note**: If you want to control what will be displayed in the summary you should override `ToString`.
+
+## If IterationSetup is provided run benchmark once per iteration
+
+When Stephen Toub says that something is [buggy](https://github.com/dotnet/BenchmarkDotNet/issues/730), it most probably is. BenchmarkDotNet performs multiple invocations of benchmark per every iteration. When we have exposed the `[IterationSetup]` attribute many users were expecting that the `IterationSetup` is going to be invoked before every benchmark execution.
+
+It was invoked before every iteration, and iteration was more than one benchmark call if the user did not configure that explicitly. We have changed that and now if you provide an `[IterationSetup]` method it is going to be executed before every iteration and iteration will invoke the benchmark just once. 
+
+```cs
+public class Test
+{
+    public static void Main() => BenchmarkRunner.Run<Test>();
+
+    [IterationSetup]
+    public void MySetup() => Console.WriteLine("MySetup");
+
+    [Benchmark]
+    public void MyBenchmark() => Console.WriteLine("MyBenchmark");
+}
+```
+
+Before:
+
+```log
+MySetup
+MyBenchmark
+MyBenchmark
+MyBenchmark
+MyBenchmark
+(...)
+```
+
+After:
+
+```log
+MySetup
+MyBenchmark
+MySetup
+MyBenchmark
+MySetup
+MyBenchmark
+(...)
+```
+
+**Note:** If you want to configure how many times benchmark should be invoked per iteration you can use the new `[InvocationCountAttribute]`.
+
+## Job Mutators
+
+`Job` represents a set of settings to run the benchmarks. We run every benchmark for every job defined by the user. The problem was that so far many jobs were just added to the config instead of being merged with other jobs.
+
+An example:
+
+```cs
+[ClrJob, CoreJob]
+[GcServer(true)]
+public class MyBenchmarkClass
+```
+
+Resulted in 3 jobs and 3 benchmark executions: `ClrJob`,  `CoreJob` and `GcServer(true)` for current runtime.
+
+Now all Jobs and their corresponding attributes marked as mutators are going to be applied to other jobs, not just added to the config. So in this particular scenario, the benchmarks from `MyBenchmarkClass` are going to be executed for .NET with Server GC enabled and .NET Core with Server GC enabled.
+
+Mutators are great when you want to have a single, global config for all benchmarks and apply given settings only to selected types. You can find out more about mutators [here](https://github.com/dotnet/BenchmarkDotNet/pull/800).
+
+## Ctrl+C
+
+When the user:
+
+* presses `Ctrl+C`
+* presses `Ctrl+Break`
+* logs off
+* closes console window
+
+We are now going to close any existing ETW session created by BenchmarkDotNet and **restore console colors** ([read more](https://github.com/dotnet/BenchmarkDotNet/pull/761)).
+
+## Handle OutOfMemoryException more gracefully
+
+When our benchmark hits `OutOfMemoryException` we print some nice explanation:
+
+```cs
+public class Program
+{
+    static void Main(string[] args) => BenchmarkRunner.Run<Program>();
+
+    private List<object> list = new List<object>();
+
+    [Benchmark]
+    public void AntiPattern() => list.Add(new int[int.MaxValue / 2]);
+}
+```
+
+```log
+OutOfMemoryException!
+BenchmarkDotNet continues to run additional iterations until desired accuracy level is achieved. It's possible only if the benchmark method doesn't have any side-effects.
+If your benchmark allocates memory and keeps it alive, you are creating a memory leak.
+You should redesign your benchmark and remove the side-effects. You can use `OperationsPerInvoke`, `IterationSetup` and `IterationCleanup` to do that.
+```
+
+## Trimming long strings
+
+We used to display the values "as is" which was bad for long strings. Now the values are trimmed ([more](https://github.com/dotnet/BenchmarkDotNet/issues/748)).
+
+```cs
+public class Long
+{
+    [Params("text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7")]
+    public string Text;
+
+    [Benchmark]
+    public int HashCode() => Text.GetHashCode();
+}
+```
+
+|   Method |                 Text |
+|--------- |--------------------- |
+| HashCode | text/(...)q=0.7 [86] |
+
+# More features
+
+## Generic benchmarks
+
+BenchmarkDotNet supports generic benchmarks, all you need to do is to tell it which types should be used as generic arguments ([read more](https://github.com/dotnet/BenchmarkDotNet/pull/758)).
+
+```cs
+[GenericTypeArguments(typeof(int))]
+[GenericTypeArguments(typeof(char))]
+public class IntroGenericTypeArguments<T>
+{
+    [Benchmark] public T Create() => Activator.CreateInstance<T>();
+}
+```
+
+## Arguments
+
+We now support more scenarios for passing arguments to benchmarks:
+
+* passing arguments to asynchronous benchmarks ([more](https://github.com/dotnet/BenchmarkDotNet/issues/818))
+* passing generic types
+* passing arguments by reference
+* passing jagged arrays ([more](https://github.com/dotnet/BenchmarkDotNet/issues/769))
+* types with implicit cast operator to stack only types can be passed as given stack-only types to Benchmarks ([more](https://github.com/dotnet/BenchmarkDotNet/issues/774))
+
+Example:
+
+```cs
+public class WithStringToReadOnlySpan
+{
+    [Benchmark]
+    [Arguments("some string")]
+    public void AcceptsReadOnlySpan(ReadOnlySpan<char> notString)
+}
+```
+
+## Console Arguments
+
+`BenchmarkSwitcher` supports various console arguments ([PR](https://github.com/dotnet/BenchmarkDotNet/pull/824)), to make it work you need to pass the `args` to switcher:
+
+```cs
+class Program
+{
+    static void Main(string[] args) 
+        => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
+}
+```
+
+**Note:** to get the most up-to-date info about supported console arguments run the benchmarks with `--help`.
+
+### Filter
+
+The `--filter` or just `-f` allows you to filter the benchmarks by their full name (`namespace.typeName.methodName`) using glob patterns.
+
+Examples:
+
+1. Run all benchmarks from System.Memory namespace: `-f System.Memory*`
+2. Run all benchmarks: `-f *`
+3. Run all benchmarks from ClassA and ClassB `-f *ClassA* *ClassB*`
+
+**Note**: If you would like to **join** all the results into a **single summary**, you need to us `--join`.
+
+### Categories
+
+You can also filter the benchmarks by categories:
+
+* `--anyCategories` - runs all benchmarks that belong to **any** of the provided categories
+* `--allCategories`- runs all benchmarks that belong to **all** provided categories
+
+### Diagnosers
+
+* `-m`, `--memory` - enables MemoryDiagnoser and prints memory statistics
+* `-d`, `--disassm`- enables DisassemblyDiagnoser and exports diassembly of benchmarked code
+
+### Runtimes
+
+The `--runtimes` or just `-r` allows you to run the benchmarks for selected Runtimes. Available options are: Clr, Mono, Core and CoreRT.
+
+Example: run the benchmarks for .NET and .NET Core:
+
+```log
+dotnet run -c Release -- --runtimes clr core
+```
+
+### More arguments
+
+* `-j`, `--job` (Default: Default) Dry/Short/Medium/Long or Default
+* `-e`, `--exporters` GitHub/StackOverflow/RPlot/CSV/JSON/HTML/XML
+* `-i`, `--inProcess` (Default: false) Run benchmarks in Process
+* `-a`, `--artifacts` Valid path to accessible directory
+* `--outliers` (Default: OnlyUpper) None/OnlyUpper/OnlyLower/All
+* `--affinity` Affinity mask to set for the benchmark process
+* `--allStats` (Default: false) Displays all statistics (min, max & more)
+* `--attribute` Run all methods with given attribute (applied to class or method)