Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use BenchmarkDotNet and Compile YamlDotNet with Optymize option in Release-* configurations #356

Merged
merged 1 commit into from Dec 5, 2018

Conversation

Projects
None yet
2 participants
@wojtpl2
Copy link
Contributor

commented Oct 10, 2018

Currently YamlDotNet is compile without optimize option.

When you decompile YamlDotNet (in IlSpy) you can see DebuggableAttribute.DebuggingModes.DisableOptimizations options :

[assembly: CompilationRelaxations(8)]
[assembly: RuntimeCompatibility(WrapNonExceptionThrows = true)]
[assembly: Debuggable(DebuggableAttribute.DebuggingModes.Default | DebuggableAttribute.DebuggingModes.DisableOptimizations | DebuggableAttribute.DebuggingModes.IgnoreSymbolStoreSequencePoints | DebuggableAttribute.DebuggingModes.EnableEditAndContinue)]
[assembly: ComVisible(false)]

It happens because in this project is Release-Signed Release-Unsigned configuration instead of Release. In new csproj inherit from Release configuration doesn't work. There is information about it:
https://www.pedrolamas.com/2017/04/24/creating-custom-build-configurations-for-the-dotnet-core-project-format/

I've added this code in csproj:

  <PropertyGroup Condition=" '$(Configuration)' == 'Release-Signed' Or '$(Configuration)' == 'Release-Unsigned' ">
    <DefineConstants>$(DefineConstants);RELEASE;TRACE</DefineConstants>
    <DebugSymbols>false</DebugSymbols>
    <DebugType>portable</DebugType>
    <Optimize>true</Optimize>
  </PropertyGroup>

I've used also BenchmarkDotNet to prove it. BenchmarkDotNet in default configuration throw errors about non-optimized dll. This is also visible in performance:

-----------------------------------------------------------------------------------
| Serialize                                                                       |
-----------------------------------------------------------------------------------
|         |      Mean |     Error |    StdDev |      Gen0 |      Gen1 | Allocated |
-----------------------------------------------------------------------------------
| v1.2.1  |  128.7 us |  1.285 us |  1.202 us |    5.8594 |    0.2441 |  23.66 KB |
| v2.2.0  |  240.6 us |  3.467 us |  3.243 us |   18.0664 |    0.4883 |  60.03 KB |
| v2.3.0  |  307.9 us |  6.112 us |  10.21 us |   20.0195 |    0.4883 |  67.32 KB |
| v3.8.0  |  292.2 us |  4.225 us |  3.952 us |   21.4844 |    0.4883 |  70.82 KB |
| v4.0.0  |  283.2 us |  3.075 us |  2.876 us |   22.9492 |    0.4883 |  74.26 KB |
| v5.2.1  |  539.5 us |  5.710 us |  5.062 us |    8.7891 |    0.9766 |  30.82 KB |
| vlatest |  145.8 us |  1.671 us |  1.563 us |    8.3008 |    0.4883 |   30.7 KB |
-----------------------------------------------------------------------------------

I've added YamlDotNet.PerformanceTests.v5.2.1 project with current avaliable YamlDotNet version.
Each YamlDotNet.PerformanceTests.v* currently use BenchmarkDotNet. You can run each test program separately e.g. by \PerformanceTests\YamlDotNet.PerformanceTests.vlatest\bin\Release\YamlDotNet.PerformanceTests.vlatest.exe. After that you will see logs from BenchmarkDotNet. It gives a lot of information:

Mean = 539.4795 us, StdErr = 0.8531 us (0.16%); N = 14, StdDev = 3.1922 us
Min = 534.9372 us, Q1 = 537.2583 us, Median = 539.9804 us, Q3 = 542.7781 us, Max = 544.3074 us
IQR = 5.5199 us, LowerFence = 528.9784 us, UpperFence = 551.0580 us
ConfidenceInterval = [535.8785 us; 543.0805 us] (CI 99.9%), Margin = 3.6010 us (0.67% of Mean)
Skewness = -0.08, Kurtosis = 1.46, MValue = 2

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\YamlDotNet.PerformanceTests.v5_2_1.ReceiptTest-report.csv
  BenchmarkDotNet.Artifacts\results\YamlDotNet.PerformanceTests.v5_2_1.ReceiptTest-report-github.md
  BenchmarkDotNet.Artifacts\results\YamlDotNet.PerformanceTests.v5_2_1.ReceiptTest-report.html

// * Detailed results *
ReceiptTest.'Serialize v5.2.1': DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3163.0; GC = Concurrent Workstation
Mean = 539.4795 us, StdErr = 0.8531 us (0.16%); N = 14, StdDev = 3.1922 us
Min = 534.9372 us, Q1 = 537.2583 us, Median = 539.9804 us, Q3 = 542.7781 us, Max = 544.3074 us
IQR = 5.5199 us, LowerFence = 528.9784 us, UpperFence = 551.0580 us
ConfidenceInterval = [535.8785 us; 543.0805 us] (CI 99.9%), Margin = 3.6010 us (0.67% of Mean)
Skewness = -0.08, Kurtosis = 1.46, MValue = 2
-------------------- Histogram --------------------
[533.778 us ; 545.466 us) | @@@@@@@@@@@@@@
---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.11.1, OS=Windows 10.0.16299.665 (1709/FallCreatorsUpdate/Redstone3)
Intel Core i7-4810MQ CPU 2.80GHz (Haswell), 1 CPU, 8 logical and 4 physical cores
Frequency=2728062 Hz, Resolution=366.5606 ns, Timer=TSC
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3163.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.3163.0


             Method |     Mean |    Error |   StdDev |  Gen 0 |  Gen 1 | Allocated |
------------------- |---------:|---------:|---------:|-------:|-------:|----------:|
 'Serialize v5.2.1' | 539.5 us | 3.601 us | 3.192 us | 8.7891 | 0.9766 |  30.82 KB |

// * Legends *
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  StdDev    : Standard deviation of all measurements
  Gen 0     : GC Generation 0 collects per 1k Operations
  Gen 1     : GC Generation 1 collects per 1k Operations
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 us      : 1 Microsecond (0.000001 sec)

// * Diagnostic Output - MemoryDiagnoser *

Of course I've updated YamlDotNet.PerformanceTests.Runner too. Generates a similar table as before. You can see the example above.

@aaubry

This comment has been minimized.

Copy link
Owner

commented Oct 10, 2018

Thanks, this looks good. I'll try to review this soon.

@wojtpl2

This comment has been minimized.

Copy link
Contributor Author

commented Oct 20, 2018

I want to improve it. E.g. mono should be run by BenchmarkDotNet. I want to add deserializer test too. I want to improve it in next PR if it is not a problem.

@wojtpl2

This comment has been minimized.

Copy link
Contributor Author

commented Dec 5, 2018

Bump ;)
I know that not only I'm waiting for it ;) If you have any suggestions or if I can help you, please let me know.

@aaubry aaubry merged commit 077155a into aaubry:master Dec 5, 2018

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build failed
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
@aaubry

This comment has been minimized.

Copy link
Owner

commented Dec 5, 2018

Thanks, I've merged your changes and a new release is currently building.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.