Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Windows Arm64 building in CI #3986

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add Windows Arm64 building in CI #3986

wants to merge 5 commits into from

Conversation

andrewlock
Copy link
Member

@andrewlock andrewlock commented Mar 31, 2023 •
edited
Loading

Summary of changes

  • Introduce new TargetPlatform enum to replace MSBuildTargetPlatform
  • Replace ForceWindowsArm64 boolean with a PlatformRequirement enum
  • Add a new arm64 build stage that uses a different azure scale set which includes the arm64 tooling

Reason for change

Introduce new TargetPlatform enum to replace MSBuildTargetPlatform

The existing MSBuildTargetPlatform is annoyingly limiting

  • Not an enum, so can't use in pattern matching 😛
  • Doesn't have all the values (arm64, arm64ec)
  • Can't set values outside of the standard ones (i.e. can't do BuildTracerHome -TargetPlatform "arm64")

Switching to the enum solves these, but we stick to the same naming to make it easier to revert back if support is added, and to interoperate with other built-in Nuke bits

Replace ForceWindowsArm64 boolean with a PlatformRequirement enum

So I can add Single so we can only build the platform we're interested in if we want

Add a new arm64 build stage.

We could add this to the existing windows build stage, but that significantly slows down the build, and isn't worth it while this isn't a supported platform

Add a new azure VMSS that includes the arm64 tooling

I updated the other VMSS scale set so as to not break the main pipeline in case of issues

I tried to update all the stages to use the new VMSS, but for some reason it made the tests really flaky, so currently We're only using the new VMSS for the arm64 build stage...

Test coverage

None - we're not testing any of this yet

Other details

Known existing limitations:

  • We can't build the Windows MSI for arm. Wix 3.x doesn't support arm, and Wix 4.x isn't stable yet
  • We can't run native unit tests on ARM64EC (test FX doesn't support it)
  • Integration tests need updating to support targeting arm
  • Profiler doesn't work (no support in libdatadog)
  • ASM doesn't work (no support in libdwaf)
  • Docker smoke tests almost certainly don't work (haven't tried, but seems very unlikely)
  • We're only using the new VMSS for the arm64 build, as it seems to make the integration tests more flaky otherwise

@andrewlock andrewlock requested a review from a team as a code owner March 31, 2023 13:43
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Mar 31, 2023 •
edited
Loading

Datadog Report

Branch report: andrew/windows-arm64
Commit report: 38b6b91

❌ dd-trace-dotnet: 15 Failed (0 Known Flaky), 0 New Flaky, 239563 Passed, 667 Skipped, 20m 24.57s Wall Time

❌ Failed Tests (15)

This report shows up to 5 failed tests.

  • NestedApp - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/nested/app".
 Inspecting worker process 8036
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

  • NoGac - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output "Fetching IIS application "sample/".
 Inspecting worker process 6788
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 " to contain "The Datadog.Trace assembly could not be found in the GAC. Make sure the tracer has been properly installed with the MSI.".
 ...

  • OutOfProcess - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/".
 Inspecting worker process 2876
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

  • WorkingApp - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected if(mixedRuntimes){console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/".
 Inspecting worker process 5704
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

  • WorkingApp - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected if(mixedRuntimes){console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/mixed".
 Inspecting worker process 6724
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from efacc59 to ffc5172 Compare March 31, 2023 14:15
tonyredondo
tonyredondo approved these changes Mar 31, 2023
View reviewed changes
Copy link
Member

@tonyredondo tonyredondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from 7d8d763 to c2ee23a Compare March 31, 2023 16:57
@andrewlock andrewlock requested review from a team as code owners March 31, 2023 16:57
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock changed the base branch from tony/windows-arm64-support-take-2 to master March 31, 2023 16:58
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from c2ee23a to 54e395e Compare April 3, 2023 12:23
@github-actions github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Apr 3, 2023
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from 54e395e to 38b6b91 Compare April 4, 2023 12:04
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from 38b6b91 to b3f275a Compare April 18, 2023 10:01
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Apr 18, 2023 •
edited
Loading

Datadog Report

Branch report: andrew/windows-arm64
Commit report: ed46a65

❌ dd-trace-dotnet: 17 Failed (0 Known Flaky), 0 New Flaky, 241288 Passed, 820 Skipped, 25m 9.93s Wall Time

❌ Failed Tests (17)

This report shows up to 5 failed tests.

  • HttpClient_SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.HttpMessageHandlerTests - Details

    Expand for error 
     Expected exit code: 0, actual exit code: -2146233082.

  • NamedPipesSubmitsMetrics - Datadog.Trace.ClrProfiler.IntegrationTests.RuntimeMetricsTests - Details

    Expand for error 
     Assert.Empty() Failure
 Expected: <empty>
 Actual:   [System.InvalidOperationException: Unexpected end of stream at position 199
              at Datadog.Trace.Util.ThrowHelper.ThrowInvalidOperationException(String message) in /_/tracer/src/Datadog.Trace/Util/ThrowHelper.cs:line 49
              at Datadog.Trace.TestHelpers.MockHttpParser.<>c__DisplayClass1_0.<<ReadRequest>g__GoNextChar|0>d.MoveNext() in D:\a\1\s\tracer\test\Datadog.Trace.TestHelpers\MockHttpParser.cs:line 40
           --- End of stack trace from previous location where exception was thrown ---
              at Datadog.Trace.TestHelpers.MockHttpParser.<>c__DisplayClass1_0.<<ReadRequest>g__ReadUntil|1>d.MoveNext() in D:\a\1\s\tracer\test\Datadog.Trace.TestHelpers\MockHttpParser.cs:line 49
           --- End of stack trace from previous location where exception was thrown ---
              at Datadog.Trace.TestHelpers.MockHttpParser.ReadRequest(Stream stream) in D:\a\1\s\tracer\test\Datadog.Trace.TestHelpers\MockHttpParser.cs:line 124
              at Datadog.Trace.TestHelpers.MockTracerAgent.NamedPipeAgent.HandleNamedPipeTraces(NamedPipeServerStream namedPipeServerStream, CancellationToken cancellationToken) in D:\a\1\s\tracer\test\Datadog.Trace.TestHelpers\MockTracerAgent.cs:line 1208
 ...

  • NestedApp - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/nested/app".
 Inspecting worker process 3808
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

  • NoGac - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output "Fetching IIS application "sample/".
 Inspecting worker process 3812
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 " to contain "The Datadog.Trace assembly could not be found in the GAC. Make sure the tracer has been properly installed with the MSI.".
 ...

  • OutOfProcess - Datadog.Trace.Tools.Runner.IntegrationTests.Checks.IisCheckTests - Details

    Expand for error 
     Expected console.Output to be 0, but found 1.
 Expected console.Output "Fetching IIS application "sample/".
 Inspecting worker process 8092
 Failed to detect target process runtime, assuming .NET Framework
 DD_PROFILING_ENABLED is not set, the continuous profiler is disabled.
 The native loader library is not loaded into the process
 The native tracer library is not loaded into the process
 Tracer is not loaded into the process
 The environment variable COR_PROFILER should be set to '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}' (current value: not set)
 The environment variable COR_ENABLE_PROFILING should be set to '1' (current value: not set)
 ...

@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock
Replace MSBuildTargetPlatform with TargetPlatform enum
6b51bef 
This msbuild built-in one causes us some complexity so just switch it out
Also replace the `ForceWindowsArm64` boolean with an enum so we can have more options
@andrewlock
Add a new build stage that runs the arm64ec build _only_
efdd325 
The output of this stage is uploaded as an artifact, but isn't currently published anywhere
@andrewlock
Switch scale sets (to the arm64 enabled scale set)
6599828
@andrewlock
Try fix some stuff
a9995ad
@andrewlock
Revert "Switch scale sets (to the arm64 enabled scale set)"
ed46a65 
This reverts commit d55397fcfb860293802d0a3d7634e9874373f80c.
@andrewlock andrewlock force-pushed the andrew/windows-arm64 branch from b3f275a to ed46a65 Compare April 18, 2023 12:24
@andrewlock
Copy link
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (3,019ms)  : 2943, 3094
     .   : milestone, 3019,
    master - mean (3,016ms)  : 2875, 3156
     .   : milestone, 3016,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (3,674ms)  : 3597, 3752
     .   : milestone, 3674,
    master - mean (3,673ms)  : 3589, 3757
     .   : milestone, 3673,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (3,133ms)  : 2990, 3275
     .   : milestone, 3133,
    master - mean (3,122ms)  : 2990, 3253
     .   : milestone, 3122,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (3,540ms)  : 3455, 3625
     .   : milestone, 3540,
    master - mean (3,538ms)  : 3415, 3661
     .   : milestone, 3538,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (3,100ms)  : 2966, 3234
     .   : milestone, 3100,
    master - mean (3,074ms)  : 2958, 3190
     .   : milestone, 3074,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (3,512ms)  : 3446, 3578
     .   : milestone, 3512,
    master - mean (3,512ms)  : 3418, 3606
     .   : milestone, 3512,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (188ms)  : 184, 193
     .   : milestone, 188,
    master - mean (188ms)  : 184, 193
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (944ms)  : 916, 973
     .   : milestone, 944,
    master - mean (938ms)  : 905, 971
     .   : milestone, 938,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (369ms)  : 363, 375
     .   : milestone, 369,
    master - mean (369ms)  : 361, 378
     .   : milestone, 369,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (1,087ms)  : 1055, 1120
     .   : milestone, 1087,
    master - mean (1,088ms)  : 1057, 1120
     .   : milestone, 1088,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3986) - mean (356ms)  : 352, 361
     .   : milestone, 356,
    master - mean (356ms)  : 350, 362
     .   : milestone, 356,

    section CallTarget+Inlining+NGEN
    This PR (3986) - mean (1,035ms)  : 1009, 1060
     .   : milestone, 1035,
    master - mean (1,041ms)  : 1010, 1072
     .   : milestone, 1041,

@andrewlock
Copy link
Member Author

Benchmarks Report 🐌

Benchmarks for #3986 compared to master:

  • All benchmarks have the same speed
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 618μs 223ns 862ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 798μs 205ns 768ns 0.398 0 0 3.22 KB
#3986 WriteAndFlushEnrichedTraces netcoreapp3.1 594μs 166ns 642ns 0 0 0 2.62 KB
#3986 WriteAndFlushEnrichedTraces net472 786μs 215ns 834ns 0.391 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 95.4μs 562ns 5.62μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 102μs 560ns 3.36μs 0.269 0 0 1.71 KB
master AllCycleMoreComplexBody netcoreapp3.1 268μs 1.32μs 5.46μs 0 0 0 9.14 KB
master AllCycleMoreComplexBody net472 299μs 520ns 1.87μs 1.47 0 0 9.31 KB
master ObjectExtractorSimpleBody netcoreapp3.1 175ns 0.203ns 0.787ns 0.0037 0 0 272 B
master ObjectExtractorSimpleBody net472 143ns 0.266ns 1.03ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.12μs 5.8ns 22.5ns 0.0517 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.03μs 7.18ns 25.9ns 0.617 0.00605 0 3.89 KB
#3986 AllCycleSimpleBody netcoreapp3.1 98.7μs 402ns 1.45μs 0 0 0 1.66 KB
#3986 AllCycleSimpleBody net472 101μs 250ns 969ns 0.232 0 0 1.71 KB
#3986 AllCycleMoreComplexBody netcoreapp3.1 274μs 1.39μs 6.5μs 0 0 0 9.14 KB
#3986 AllCycleMoreComplexBody net472 269μs 705ns 2.54μs 1.47 0 0 9.31 KB
#3986 ObjectExtractorSimpleBody netcoreapp3.1 174ns 0.168ns 0.65ns 0.00367 0 0 272 B
#3986 ObjectExtractorSimpleBody net472 145ns 0.271ns 1.02ns 0.0446 0 0 281 B
#3986 ObjectExtractorMoreComplexBody netcoreapp3.1 3.99μs 4.25ns 15.9ns 0.0518 0 0 3.78 KB
#3986 ObjectExtractorMoreComplexBody net472 4.14μs 6.97ns 26.1ns 0.618 0.00628 0 3.89 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 186μs 229ns 792ns 0.187 0 0 20.36 KB
master SendRequest net472 0.00144ns 0.000341ns 0.00132ns 0 0 0 0 b
#3986 SendRequest netcoreapp3.1 184μs 201ns 752ns 0.183 0 0 20.36 KB
#3986 SendRequest net472 0.000201ns 0.000113ns 0.000408ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #3986

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 42.14 KB 41.68 KB -461 B -1.09%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 616μs 1.85μs 7.17μs 0.308 0 0 42.14 KB
master WriteAndFlushEnrichedTraces net472 809μs 3.63μs 14μs 8.36 2.39 0.398 53.27 KB
#3986 WriteAndFlushEnrichedTraces netcoreapp3.1 596μs 1.03μs 3.85μs 0.305 0 0 41.68 KB
#3986 WriteAndFlushEnrichedTraces net472 792μs 3.48μs 13.5μs 8.36 2.39 0.398 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.32μs 0.748ns 2.8ns 0.0119 0 0 904 B
master ExecuteNonQuery net472 1.67μs 1.63ns 6.3ns 0.144 0.000828 0 907 B
#3986 ExecuteNonQuery netcoreapp3.1 1.28μs 0.884ns 3.31ns 0.0121 0 0 904 B
#3986 ExecuteNonQuery net472 1.77μs 2.44ns 9.43ns 0.144 0 0 907 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.34μs 0.478ns 1.79ns 0.014 0 0 1.06 KB
master CallElasticsearch net472 2.06μs 0.632ns 2.45ns 0.176 0 0 1.11 KB
master CallElasticsearchAsync netcoreapp3.1 1.43μs 0.4ns 1.5ns 0.0164 0 0 1.18 KB
master CallElasticsearchAsync net472 2.34μs 0.8ns 3.1ns 0.197 0 0 1.24 KB
#3986 CallElasticsearch netcoreapp3.1 1.34μs 0.348ns 1.26ns 0.0141 0 0 1.06 KB
#3986 CallElasticsearch net472 2.26μs 0.749ns 2.9ns 0.175 0.00114 0 1.11 KB
#3986 CallElasticsearchAsync netcoreapp3.1 1.41μs 0.611ns 2.29ns 0.0156 0 0 1.18 KB
#3986 CallElasticsearchAsync net472 2.49μs 0.53ns 1.98ns 0.197 0.00124 0 1.24 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 1.01ns 3.91ns 0.0172 0 0 1.28 KB
master ExecuteAsync net472 1.85μs 0.505ns 1.96ns 0.206 0.000926 0 1.3 KB
#3986 ExecuteAsync netcoreapp3.1 1.47μs 0.716ns 2.77ns 0.0174 0 0 1.28 KB
#3986 ExecuteAsync net472 1.84μs 0.518ns 2ns 0.206 0.000917 0 1.3 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.17μs 1.21ns 4.37ns 0.0353 0 0 2.66 KB
master SendAsync net472 6.93μs 2.13ns 8.26ns 0.479 0 0 3.03 KB
#3986 SendAsync netcoreapp3.1 4.26μs 1.39ns 5.2ns 0.036 0 0 2.66 KB
#3986 SendAsync net472 6.82μs 1.93ns 7.24ns 0.478 0 0 3.03 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.95μs 0.845ns 3.16ns 0.0244 0 0 1.83 KB
master EnrichedLog net472 2.49μs 1.34ns 5.21ns 0.284 0 0 1.79 KB
#3986 EnrichedLog netcoreapp3.1 2.01μs 0.838ns 3.13ns 0.0254 0 0 1.83 KB
#3986 EnrichedLog net472 2.52μs 1.59ns 5.93ns 0.284 0 0 1.79 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 119μs 170ns 660ns 0.0596 0 0 4.42 KB
master EnrichedLog net472 149μs 88.7ns 343ns 0.672 0.224 0 4.63 KB
#3986 EnrichedLog netcoreapp3.1 118μs 273ns 1.06μs 0 0 0 4.42 KB
#3986 EnrichedLog net472 145μs 87.7ns 340ns 0.726 0.218 0 4.63 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.81μs 2.48ns 9.59ns 0.0521 0 0 3.9 KB
master EnrichedLog net472 5.32μs 2.1ns 8.15ns 0.565 0.00265 0 3.56 KB
#3986 EnrichedLog netcoreapp3.1 3.96μs 1.78ns 6.91ns 0.0534 0 0 3.9 KB
#3986 EnrichedLog net472 4.99μs 1.4ns 5.42ns 0.565 0.00251 0 3.56 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.57μs 0.722ns 2.6ns 0.0174 0 0 1.3 KB
master SendReceive net472 1.95μs 1.78ns 6.87ns 0.212 0 0 1.34 KB
#3986 SendReceive netcoreapp3.1 1.6μs 0.413ns 1.49ns 0.0175 0 0 1.3 KB
#3986 SendReceive net472 2.02μs 1ns 3.62ns 0.213 0 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.75μs 2.39ns 8.63ns 0.0242 0 0 1.78 KB
master EnrichedLog net472 4.34μs 1.43ns 5.37ns 0.349 0 0 2.21 KB
#3986 EnrichedLog netcoreapp3.1 3.73μs 1.67ns 6.25ns 0.0242 0 0 1.78 KB
#3986 EnrichedLog net472 4.33μs 1.43ns 5.55ns 0.349 0 0 2.21 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 681ns 0.181ns 0.628ns 0.00956 0 0 720 B
master StartFinishSpan net472 866ns 0.394ns 1.53ns 0.121 0 0 762 B
master StartFinishScope netcoreapp3.1 873ns 0.539ns 2.09ns 0.0113 0 0 840 B
master StartFinishScope net472 1.07μs 0.483ns 1.81ns 0.134 0 0 842 B
#3986 StartFinishSpan netcoreapp3.1 721ns 0.208ns 0.805ns 0.00973 0 0 720 B
#3986 StartFinishSpan net472 866ns 0.257ns 0.96ns 0.121 0 0 762 B
#3986 StartFinishScope netcoreapp3.1 805ns 0.224ns 0.809ns 0.0113 0 0 840 B
#3986 StartFinishScope net472 1.14μs 1.88ns 7.29ns 0.134 0 0 842 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 928ns 1ns 3.88ns 0.0112 0 0 840 B
master RunOnMethodBegin net472 1.25μs 0.613ns 2.37ns 0.133 0 0 842 B
#3986 RunOnMethodBegin netcoreapp3.1 868ns 0.439ns 1.7ns 0.0114 0 0 840 B
#3986 RunOnMethodBegin net472 1.25μs 0.348ns 1.3ns 0.134 0 0 842 B

@andrewlock
Copy link
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!
Loading 
gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3986) (11.357M)   : 0, 11357200
    master (11.385M)   : 0, 11385161
    benchmarks/2.9.0 (11.162M)   : 0, 11161538

    section Automatic
    This PR (3986) (7.948M)   : 0, 7948082
    master (7.915M)   : 0, 7914877
    benchmarks/2.9.0 (8.099M)   : 0, 8099075

    section Trace stats
    master (8.075M)   : 0, 8074667

    section Manual
    This PR (3986) (9.588M)   : 0, 9587622
    master (9.743M)   : 0, 9742606

    section Manual + Automatic
    This PR (3986) (7.651M)   : 0, 7650730
    master (7.736M)   : 0, 7735510

    section Version Conflict
    master (6.956M)   : 0, 6956437
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3986) (9.576M)   : 0, 9576268
    master (9.472M)   : 0, 9472029
    benchmarks/2.9.0 (9.705M)   : 0, 9704781

    section Automatic
    This PR (3986) (6.876M)   : 0, 6875825
    master (6.680M)   : 0, 6679764

    section Trace stats
    master (6.894M)   : 0, 6894147

    section Manual
    This PR (3986) (8.639M)   : 0, 8638622
    master (8.600M)   : 0, 8600389

    section Manual + Automatic
    This PR (3986) (6.535M)   : 0, 6535370
    master (6.525M)   : 0, 6524620

    section Version Conflict
    master (5.872M)   : 0, 5872057
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3986) (9.567M)   : 0, 9566653
    master (10.045M)   : 0, 10044623
    benchmarks/2.29.0 (9.984M)   : 0, 9984467
    benchmarks/2.9.0 (9.827M)   : 0, 9827121

    section Automatic
    This PR (3986) (6.929M)   : 0, 6929444
    master (6.966M)   : 0, 6966417
    benchmarks/2.29.0 (7.308M)   : 0, 7308052
    benchmarks/2.9.0 (7.246M)   : 0, 7246397

    section Trace stats
    master (6.870M)   : 0, 6869583
    benchmarks/2.29.0 (7.178M)   : 0, 7177618

    section Manual
    This PR (3986) (8.629M)   : 0, 8628700
    master (8.561M)   : 0, 8561162
    benchmarks/2.29.0 (8.782M)   : 0, 8781971

    section Manual + Automatic
    This PR (3986) (6.812M)   : 0, 6812393
    master (6.853M)   : 0, 6853010
    benchmarks/2.29.0 (7.020M)   : 0, 7019745

    section Version Conflict
    master (6.037M)   : 0, 6036526
    benchmarks/2.29.0 (6.225M)   : 0, 6224998
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3986) (7.595M)   : 0, 7595273
    master (7.441M)   : 0, 7441390
    benchmarks/2.29.0 (7.542M)   : 0, 7542426
    benchmarks/2.9.0 (7.748M)   : 0, 7748029

    section No attack
    This PR (3986) (2.193M)   : 0, 2193090
    master (2.145M)   : 0, 2145382
    benchmarks/2.29.0 (2.187M)   : 0, 2187214
    benchmarks/2.9.0 (3.274M)   : 0, 3273699

    section Attack
    This PR (3986) (1.860M)   : 0, 1859648
    master (1.814M)   : 0, 1813658
    benchmarks/2.29.0 (1.860M)   : 0, 1860237
    benchmarks/2.9.0 (2.598M)   : 0, 2597950

    section Blocking
    This PR (3986) (3.754M)   : 0, 3754185
    master (3.671M)   : 0, 3671287
    benchmarks/2.29.0 (3.254M)   : 0, 3254362

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonyredondo tonyredondo

@pierotibou pierotibou

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewlock @tonyredondo @pierotibou