Add missing tests from .NET Core 3.0 perf tests blog post #478

stephentoub · 2019-05-17T05:52:42Z

https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-3-0/

cc: @adamsitnik, @billwert

billwert · 2019-05-17T18:03:33Z

/azp run

azure-pipelines · 2019-05-17T18:03:48Z

Azure Pipelines successfully started running 1 pipeline(s).

adamsitnik

@stephentoub thank you very much for adding all these benchmarks to the performance repo!

Overall LGTM, but I left a few questions.

Could you please rever the renaming changes?

src/benchmarks/micro/corefx/System.Memory/ReadOnlySpan.cs

src/benchmarks/micro/corefx/System.Net.Http/Perf.SocketsHttpHandler.cs

adamsitnik · 2019-05-17T21:47:48Z

src/benchmarks/micro/corefx/System.Net.Primitives/DnsTests.cs

+    public class DnsTests
+    {
+        [Benchmark]
+        public IPHostEntry GetHostEntry() => Dns.GetHostEntry("34.206.253.53");


what is this address pointing to? what are we measuring here?

I want to have a better understanding.

src/benchmarks/micro/corefx/System.Runtime.Extensions/Perf.StreamWriter.cs

src/benchmarks/micro/corefx/System.Runtime.Extensions/Perf.WebUtility.cs

src/benchmarks/micro/corefx/System.Runtime/Perf.Array.cs

src/benchmarks/micro/corefx/System.Runtime/Perf.Char.cs

src/benchmarks/micro/corefx/System.Runtime/Perf.Decimal.cs

adamsitnik · 2019-05-17T22:01:30Z

src/benchmarks/micro/corefx/System.Runtime/Perf.Int32.cs

@@ -17,35 +18,45 @@ public class Perf_Int32
        public static IEnumerable<object> Values => new object[]
        {
            int.MinValue,
+            4, // single digit


should we also add such a test case for uint, long and ulong?

adamsitnik · 2019-05-17T22:04:03Z

src/benchmarks/micro/corefx/System.Threading.Timers/Perf.Timer.cs

+            {
+                _timers[i] = new Timer(_ => { }, null, i, i);
+            }
+            Thread.Sleep(1000);


nit: it would be nice to add a comment explaining why we need this sleep here

# Conflicts: # src/benchmarks/micro/corefx/System.Linq/Perf.Linq.cs # src/benchmarks/micro/corefx/System.Runtime/Perf.Guid.cs

… solving merge conflict

adamsitnik · 2019-07-18T10:27:33Z

@stephentoub I am currently doing the 2.2 vs 3.0 comparison and I really want to include these benchmarks in my results. To save some time I've just solved the merged conflict in your branch and fixed the few issues that I've found during the review. I hope you don't mind.

stephentoub · 2019-07-18T12:34:38Z

I hope you don't mind.

That's fine, thanks.

michellemcdaniel · 2019-07-18T16:54:44Z

It looks like this change is causing build failures in the the lab. See: https://dev.azure.com/dnceng/internal/_build/results?buildId=270482

What I am seeing is this:

[2019/07/18 09:05:08][INFO] // **************************
[2019/07/18 09:05:08][INFO] // Benchmark: Perf_StreamWriter.WriteFormat: Job-VRJOQN(Runtime=Core, Toolchain=netcoreapp3.0, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2019/07/18 09:05:08][INFO] // *** Execute ***
[2019/07/18 09:05:08][INFO] // Launch: 1 / 1
[2019/07/18 09:05:08][INFO] // Execute: dotnet "b436a215-f884-47e8-95e3-17588bea18ec.dll" --benchmarkName "System.IO.Tests.Perf_StreamWriter.WriteFormat" --job "Runtime=Core, Toolchain=netcoreapp3.0, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1" --benchmarkId 267 in C:\dotnetbuild\work\c9ccec99-5f20-4090-97a6-47555d130716\Payload\artifacts\bin\MicroBenchmarks\Release\netcoreapp3.0\b436a215-f884-47e8-95e3-17588bea18ec\bin\Release\netcoreapp3.0
[2019/07/18 09:05:09][INFO] // BeforeAnythingElse
[2019/07/18 09:05:09][INFO] 
[2019/07/18 09:05:09][INFO] // Benchmark Process Environment Information:
[2019/07/18 09:05:09][INFO] // Runtime=.NET Core 3.0.0-preview8-27917-07 (CoreCLR 4.700.19.36603, CoreFX 4.700.19.36703), 64bit RyuJIT
[2019/07/18 09:05:09][INFO] // GC=Concurrent Workstation
[2019/07/18 09:05:09][INFO] // Job: Job-YRPDCT(Runtime=Core, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2019/07/18 09:05:09][INFO] 
[2019/07/18 09:05:09][INFO] OverheadJitting  1: 1 op, 214113.04 ns, 214.1130 us/op
[2019/07/18 09:05:09][INFO] WorkloadJitting  1: 1 op, 327299.30 ns, 327.2993 us/op
[2019/07/18 09:05:09][INFO] 
[2019/07/18 09:05:09][INFO] OverheadJitting  2: 16 op, 205845.03 ns, 12.8653 us/op
[2019/07/18 09:05:09][INFO] WorkloadJitting  2: 16 op, 212117.32 ns, 13.2573 us/op
[2019/07/18 09:05:09][INFO] 
[2019/07/18 09:05:09][INFO] WorkloadPilot    1: 16 op, 5987.18 ns, 374.1989 ns/op
[2019/07/18 09:05:09][INFO] WorkloadPilot    2: 668096 op, 143038061.66 ns, 214.0981 ns/op
[2019/07/18 09:05:09][INFO] WorkloadPilot    3: 1167696 op, 237505249.48 ns, 203.3965 ns/op
[2019/07/18 09:05:09][INFO] WorkloadPilot    4: 1229136 op, 247925512.61 ns, 201.7071 ns/op
[2019/07/18 09:05:10][INFO] WorkloadPilot    5: 1239424 op, 249375266.04 ns, 201.2025 ns/op
[2019/07/18 09:05:10][INFO] WorkloadPilot    6: 1242544 op, 252487460.42 ns, 203.2020 ns/op
[2019/07/18 09:05:10][INFO] WorkloadPilot    7: 1230304 op, 247871057.76 ns, 201.4714 ns/op
[2019/07/18 09:05:10][INFO] WorkloadPilot    8: 1240880 op, 249637276.54 ns, 201.1776 ns/op
[2019/07/18 09:05:11][INFO] WorkloadPilot    9: 1242688 op, 250029864.64 ns, 201.2008 ns/op
[2019/07/18 09:05:11][INFO] WorkloadPilot   10: 1242544 op, 249938346.28 ns, 201.1505 ns/op
[2019/07/18 09:05:11][INFO] WorkloadPilot   11: 1242864 op, 250023877.45 ns, 201.1675 ns/op
[2019/07/18 09:05:11][INFO] 
[2019/07/18 09:05:11][INFO] OverheadWarmup   1: 1242864 op, 2042199.37 ns, 1.6431 ns/op
[2019/07/18 09:05:11][INFO] OverheadWarmup   2: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadWarmup   3: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadWarmup   4: 1242864 op, 1881685.87 ns, 1.5140 ns/op
[2019/07/18 09:05:11][INFO] OverheadWarmup   5: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadWarmup   6: 1242864 op, 1881685.87 ns, 1.5140 ns/op
[2019/07/18 09:05:11][INFO] 
[2019/07/18 09:05:11][INFO] OverheadActual   1: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   2: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   3: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   4: 1242864 op, 1903068.66 ns, 1.5312 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   5: 1242864 op, 1881685.87 ns, 1.5140 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   6: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   7: 1242864 op, 1881685.87 ns, 1.5140 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   8: 1242864 op, 1881685.87 ns, 1.5140 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual   9: 1242864 op, 1882826.28 ns, 1.5149 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  10: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  11: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  12: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  13: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  14: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] OverheadActual  15: 1242864 op, 1881970.97 ns, 1.5142 ns/op
[2019/07/18 09:05:11][INFO] 
[2019/07/18 09:05:11][INFO] WorkloadWarmup   1: 1242864 op, 250334355.62 ns, 201.4173 ns/op
[2019/07/18 09:05:11][INFO] 
[2019/07/18 09:05:11][INFO] // BeforeActualRun
[2019/07/18 09:05:12][INFO] WorkloadActual   1: 1242864 op, 250122523.41 ns, 201.2469 ns/op
[2019/07/18 09:05:12][INFO] WorkloadActual   2: 1242864 op, 250557021.78 ns, 201.5965 ns/op
[2019/07/18 09:05:12][INFO] WorkloadActual   3: 1242864 op, 252192948.07 ns, 202.9127 ns/op
[2019/07/18 09:05:12][INFO] WorkloadActual   4: 1242864 op, 250592659.77 ns, 201.6252 ns/op
[2019/07/18 09:05:13][INFO] WorkloadActual   5: 1242864 op, 253547191.68 ns, 204.0024 ns/op
[2019/07/18 09:05:13][INFO] WorkloadActual   6: 1242864 op, 253022030.26 ns, 203.5798 ns/op
[2019/07/18 09:05:13][INFO] WorkloadActual   7: 1242864 op, 250249109.55 ns, 201.3487 ns/op
[2019/07/18 09:05:13][INFO] WorkloadActual   8: 1242864 op, 252046119.55 ns, 202.7946 ns/op
[2019/07/18 09:05:14][INFO] WorkloadActual   9: 1242864 op, 250376836.10 ns, 201.4515 ns/op
[2019/07/18 09:05:14][INFO] WorkloadActual  10: 1242864 op, 253538638.57 ns, 203.9955 ns/op
[2019/07/18 09:05:14][INFO] WorkloadActual  11: 1242864 op, 250251675.48 ns, 201.3508 ns/op
[2019/07/18 09:05:14][INFO] WorkloadActual  12: 1242864 op, 252248543.33 ns, 202.9575 ns/op
[2019/07/18 09:05:15][INFO] WorkloadActual  13: 1242864 op, 250186101.58 ns, 201.2981 ns/op
[2019/07/18 09:05:15][INFO] WorkloadActual  14: 1242864 op, 253024881.30 ns, 203.5821 ns/op
[2019/07/18 09:05:15][INFO] WorkloadActual  15: 1242864 op, 250351176.75 ns, 201.4309 ns/op
[2019/07/18 09:05:15][INFO] 
[2019/07/18 09:05:15][INFO] // AfterActualRun
[2019/07/18 09:05:15][INFO] WorkloadResult   1: 1242864 op, 248239145.93 ns, 199.7315 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   2: 1242864 op, 248673644.30 ns, 200.0811 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   3: 1242864 op, 250309570.59 ns, 201.3974 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   4: 1242864 op, 248709282.29 ns, 200.1098 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   5: 1242864 op, 251663814.20 ns, 202.4870 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   6: 1242864 op, 251138652.78 ns, 202.0645 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   7: 1242864 op, 248365732.07 ns, 199.8334 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   8: 1242864 op, 250162742.07 ns, 201.2793 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult   9: 1242864 op, 248493458.62 ns, 199.9362 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  10: 1242864 op, 251655261.08 ns, 202.4801 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  11: 1242864 op, 248368298.00 ns, 199.8355 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  12: 1242864 op, 250365165.85 ns, 201.4421 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  13: 1242864 op, 248302724.10 ns, 199.7827 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  14: 1242864 op, 251141503.82 ns, 202.0668 ns/op
[2019/07/18 09:05:15][INFO] WorkloadResult  15: 1242864 op, 248467799.27 ns, 199.9155 ns/op
[2019/07/18 09:05:15][INFO] GC:  7 0 0 29828736 1242864
[2019/07/18 09:05:15][INFO] 
[2019/07/18 09:05:15][INFO] // ERROR:  Exception during GlobalCleanup!
[2019/07/18 09:05:15][INFO] // ERROR:  Cannot access a closed Stream.
[2019/07/18 09:05:15][INFO] // AfterAll
[2019/07/18 09:05:15][INFO] Exception during GlobalCleanup!
[2019/07/18 09:05:15][INFO] Cannot access a closed Stream.
[2019/07/18 09:05:15][INFO] 
[2019/07/18 09:05:15][INFO] Mean = 200.8295 ns, StdErr = 0.2796 ns (0.14%); N = 15, StdDev = 1.0830 ns
[2019/07/18 09:05:15][INFO] Min = 199.7315 ns, Q1 = 199.8355 ns, Median = 200.1098 ns, Q3 = 202.0645 ns, Max = 202.4870 ns
[2019/07/18 09:05:15][INFO] IQR = 2.2290 ns, LowerFence = 196.4919 ns, UpperFence = 205.4080 ns
[2019/07/18 09:05:15][INFO] ConfidenceInterval = [199.6717 ns; 201.9873 ns] (CI 99.9%), Margin = 1.1578 ns (0.58% of Mean)
[2019/07/18 09:05:15][INFO] Skewness = 0.35, Kurtosis = 1.28, MValue = 2

Which leads to failures when we try to write measurement.json later:

[2019/07/18 09:34:16][INFO] $ C:\Python37\python.exe C:\dotnetbuild\work\c9ccec99-5f20-4090-97a6-47555d130716\Payload\tools\Microsoft.BenchView.JSONFormat\tools\measurement.py bdn --append results\System.IO.Tests.Perf_StreamWriter-report-full.json
[2019/07/18 09:34:16][INFO] [2019-07-18 09:34][ERROR] Unexpected exception caught: 'BenchmarkDotNet' object has no attribute 'Metrics'
[2019/07/18 09:34:16][INFO] [2019-07-18 09:34][ERROR] Traceback (most recent call last):
[2019/07/18 09:34:16][INFO]   File "C:\dotnetbuild\work\c9ccec99-5f20-4090-97a6-47555d130716\Payload\tools\Microsoft.BenchView.JSONFormat\tools\measurement.py", line 302, in main
[2019/07/18 09:34:16][INFO]     merge_tests(tests, infile, args)
[2019/07/18 09:34:16][INFO]   File "C:\dotnetbuild\work\c9ccec99-5f20-4090-97a6-47555d130716\Payload\tools\Microsoft.BenchView.JSONFormat\tools\measurement.py", line 270, in merge_tests
[2019/07/18 09:34:16][INFO]     for test_names, value, metric_info in testDataGenerator:
[2019/07/18 09:34:16][INFO]   File "C:\dotnetbuild\work\c9ccec99-5f20-4090-97a6-47555d130716\Payload\tools\Microsoft.BenchView.JSONFormat\tools\benchview\tools\measurement_bdn.py", line 70, in read_test_data
[2019/07/18 09:34:16][INFO]     for bdnmetric in b.Metrics:
[2019/07/18 09:34:16][INFO] AttributeError: 'BenchmarkDotNet' object has no attribute 'Metrics'
[2019/07/18 09:34:16][INFO] 
[2019/07/18 09:34:16][ERROR] Process exited with status 1

How necessary is the WriteFormat perf test, and can we back out that single test to get the lab clean?

adamsitnik · 2019-07-18T16:55:53Z

@adiaaida I am going to send a fix in a few minutes

michellemcdaniel · 2019-07-18T16:56:04Z

Great, thank you.

billwert · 2019-07-18T17:01:13Z

Hm. I think CI would have caught this, but I only see the WIP check. We should understand where the rest went.

michellemcdaniel · 2019-07-18T17:04:10Z

AzDO was down yesterday afternoon/evening for PRs (they weren't triggering)

billwert · 2019-07-18T17:09:00Z

Ah, got it.

Add missing tests from .NET Core 3.0 perf tests blog post

e44a321

adamsitnik requested changes May 17, 2019

View reviewed changes

adamsitnik mentioned this pull request May 27, 2019

update BenchmarkDotNet to latest version to take advantage of improvements #511

Merged

adamsitnik added 7 commits July 18, 2019 11:45

Merge remote-tracking branch 'upstream/master' into netcore3perftests

8229c20

# Conflicts: # src/benchmarks/micro/corefx/System.Linq/Perf.Linq.cs # src/benchmarks/micro/corefx/System.Runtime/Perf.Guid.cs

revert the renames that would change benchmark ID

d287b0d

split ToFromBytes to two separate benchmarks that measure 1 thing

8840219

fix the TrimArguments (IEnumerable<object[]> => IEnumerable<object>)

ff7c061

revert StreamWriter changes

2134834

fix StreamWriter

7c03aa4

bring back EmptyTakeSelectToArray which I've accidentaly removed when…

40e3d78

… solving merge conflict

adamsitnik approved these changes Jul 18, 2019

View reviewed changes

adamsitnik merged commit 7d96fd3 into dotnet:master Jul 18, 2019

stephentoub deleted the netcore3perftests branch July 18, 2019 12:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add missing tests from .NET Core 3.0 perf tests blog post #478

Add missing tests from .NET Core 3.0 perf tests blog post #478

stephentoub commented May 17, 2019

billwert commented May 17, 2019

azure-pipelines bot commented May 17, 2019

adamsitnik left a comment

adamsitnik May 17, 2019

adamsitnik May 17, 2019

adamsitnik May 17, 2019

adamsitnik commented Jul 18, 2019

stephentoub commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

adamsitnik commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

billwert commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

billwert commented Jul 18, 2019

Add missing tests from .NET Core 3.0 perf tests blog post #478

Add missing tests from .NET Core 3.0 perf tests blog post #478

Conversation

stephentoub commented May 17, 2019

billwert commented May 17, 2019

azure-pipelines bot commented May 17, 2019

adamsitnik left a comment

Choose a reason for hiding this comment

adamsitnik May 17, 2019

Choose a reason for hiding this comment

adamsitnik May 17, 2019

Choose a reason for hiding this comment

adamsitnik May 17, 2019

Choose a reason for hiding this comment

adamsitnik commented Jul 18, 2019

stephentoub commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

adamsitnik commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

billwert commented Jul 18, 2019

michellemcdaniel commented Jul 18, 2019

billwert commented Jul 18, 2019