Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flake in telemetry metric unit test #4622

Merged
merged 1 commit into from
Sep 14, 2023
Merged

Fix flake in telemetry metric unit test #4622

merged 1 commit into from
Sep 14, 2023

Conversation

andrewlock
Copy link
Member

Summary of changes

  • Add check that aggregation task hasn't faulted in unit tests
  • Remove timing requirement from test to reduce flake

Reason for change

On ARM64 the test that verifies the aggregation task runs periodically was flaky. This is likely down to the ci machines being overloaded, but it was annoying. Kevin also pointed out that we couldn't be sure that this was the case, and it was possible that that the aggregation task had faulted, so added await to the task in all unit tests

Implementation details

Added an optional "aggregation notification" that we can use in unit tests to wait for an aggregation to occur. Also added await collector.DisposeAsync() to all unit tests.

Test coverage

Covered by existing

@andrewlock
Fix flaky test
ee7d35c 
- Ensure we await collector disposal to rethrow any exceptions
- Remove timing requirement
@andrewlock andrewlock added the area:tests unit tests, integration tests label Sep 14, 2023
@andrewlock andrewlock requested a review from a team as a code owner September 14, 2023 10:06
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Sep 14, 2023 •
edited
Loading

Datadog Report

Branch report: andrew/ci/flaky-metrics
Commit report: ee7d35c

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 302173 Passed, 1154 Skipped, 22m 42.86s Wall Time

@andrewlock
Copy link
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (68ms)  : 65, 71
     .   : milestone, 68,
    master - mean (71ms)  : 58, 83
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (1,035ms)  : 1012, 1058
     .   : milestone, 1035,
    master - mean (1,035ms)  : 1008, 1062
     .   : milestone, 1035,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (104ms)  : 101, 108
     .   : milestone, 104,
    master - mean (105ms)  : 97, 114
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (750ms)  : 730, 770
     .   : milestone, 750,
    master - mean (747ms)  : 731, 762
     .   : milestone, 747,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (88ms)  : 85, 92
     .   : milestone, 88,
    master - mean (89ms)  : 84, 93
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (708ms)  : 683, 734
     .   : milestone, 708,
    master - mean (711ms)  : 694, 728
     .   : milestone, 711,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (185ms)  : 174, 196
     .   : milestone, 185,
    master - mean (186ms)  : 179, 193
     .   : milestone, 186,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (1,122ms)  : 1084, 1160
     .   : milestone, 1122,
    master - mean (1,126ms)  : 1097, 1155
     .   : milestone, 1126,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (264ms)  : 255, 274
     .   : milestone, 264,
    master - mean (271ms)  : 261, 281
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (1,083ms)  : 1051, 1114
     .   : milestone, 1083,
    master - mean (1,094ms)  : 1062, 1125
     .   : milestone, 1094,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4622) - mean (256ms)  : 243, 268
     .   : milestone, 256,
    master - mean (257ms)  : 244, 269
     .   : milestone, 257,

    section CallTarget+Inlining+NGEN
    This PR (4622) - mean (1,033ms)  : 1001, 1066
     .   : milestone, 1033,
    master - mean (1,048ms)  : 1014, 1083
     .   : milestone, 1048,
Loading

@andrewlock
Copy link
Member Author

Benchmarks Report 🐌

Benchmarks for #4622 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.142
  • 5 benchmarks are slower, with geometric mean 1.194
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.12μs 45.3ns 283ns 0.0241 0.0121 0 7.29 KB
master StartStopWithChild netcoreapp3.1 10.2μs 54.2ns 282ns 0.0294 0.00979 0 7.39 KB
master StartStopWithChild net472 15.6μs 53ns 205ns 1.25 0.281 0.0759 7.66 KB
#4622 StartStopWithChild net6.0 8.2μs 46.3ns 331ns 0.0244 0.0122 0 7.29 KB
#4622 StartStopWithChild netcoreapp3.1 9.8μs 50.4ns 252ns 0.0299 0.015 0 7.39 KB
#4622 StartStopWithChild net472 15.5μs 44.2ns 171ns 1.27 0.322 0.0996 7.66 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 480μs 118ns 456ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 661μs 580ns 2.25μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 792μs 295ns 1.14μs 0.393 0 0 3.3 KB
#4622 WriteAndFlushEnrichedTraces net6.0 484μs 444ns 1.72μs 0 0 0 2.7 KB
#4622 WriteAndFlushEnrichedTraces netcoreapp3.1 639μs 293ns 1.14μs 0 0 0 2.7 KB
#4622 WriteAndFlushEnrichedTraces net472 772μs 194ns 700ns 0.386 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4622

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.449 115.40 167.18

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 42.2μs 20.2ns 78.4ns 0.0214 0 0 1.78 KB
master AllCycleSimpleBody netcoreapp3.1 45.6μs 79.4ns 308ns 0.0229 0 0 1.75 KB
master AllCycleSimpleBody net472 47.4μs 38ns 147ns 0.283 0 0 1.82 KB
master AllCycleMoreComplexBody net6.0 235μs 98.6ns 382ns 0.118 0 0 9.35 KB
master AllCycleMoreComplexBody netcoreapp3.1 250μs 423ns 1.64μs 0.123 0 0 9.24 KB
master AllCycleMoreComplexBody net472 263μs 82.4ns 319ns 1.44 0 0 9.42 KB
master ObjectExtractorSimpleBody net6.0 115ns 0.0422ns 0.158ns 0.00392 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 173ns 0.0821ns 0.318ns 0.00366 0 0 272 B
master ObjectExtractorSimpleBody net472 144ns 0.104ns 0.403ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.08μs 1.12ns 4.32ns 0.0539 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4μs 1.97ns 7.64ns 0.0501 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.17μs 2.9ns 10.9ns 0.617 0.00623 0 3.89 KB
#4622 AllCycleSimpleBody net6.0 43.4μs 24.9ns 96.3ns 0.0216 0 0 1.78 KB
#4622 AllCycleSimpleBody netcoreapp3.1 44.9μs 143ns 536ns 0.0226 0 0 1.75 KB
#4622 AllCycleSimpleBody net472 48.9μs 71.9ns 278ns 0.288 0 0 1.82 KB
#4622 AllCycleMoreComplexBody net6.0 239μs 276ns 1.03μs 0.12 0 0 9.35 KB
#4622 AllCycleMoreComplexBody netcoreapp3.1 249μs 291ns 1.62μs 0 0 0 9.24 KB
#4622 AllCycleMoreComplexBody net472 265μs 67.7ns 253ns 1.45 0 0 9.42 KB
#4622 ObjectExtractorSimpleBody net6.0 167ns 0.0302ns 0.109ns 0.00389 0 0 280 B
#4622 ObjectExtractorSimpleBody netcoreapp3.1 171ns 0.0638ns 0.247ns 0.0037 0 0 272 B
#4622 ObjectExtractorSimpleBody net472 145ns 0.135ns 0.522ns 0.0446 0 0 281 B
#4622 ObjectExtractorMoreComplexBody net6.0 3.02μs 0.612ns 2.29ns 0.0543 0 0 3.88 KB
#4622 ObjectExtractorMoreComplexBody netcoreapp3.1 4.09μs 2ns 7.47ns 0.051 0 0 3.78 KB
#4622 ObjectExtractorMoreComplexBody net472 4.12μs 3.01ns 11.7ns 0.617 0.0062 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 51.9μs 16.2ns 60.6ns 0.207 0 0 16.07 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 69.9μs 189ns 730ns 0.193 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 93.7μs 22.1ns 82.5ns 2.53 0.0936 0 16.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 113μs 40.4ns 156ns 0.298 0 0 22.42 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 138μs 473ns 1.83μs 0.282 0 0 22.37 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 165μs 88.1ns 329ns 3.54 0.164 0 22.71 KB
master RunWaf(args=NestedMap (100)) net6.0 118μs 83.7ns 324ns 0.472 0 0 34.74 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 154μs 145ns 504ns 0.482 0 0 35.39 KB
master RunWaf(args=NestedMap (100)) net472 214μs 146ns 528ns 5.67 0.428 0 36.34 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 199μs 444ns 1.6μs 0.492 0 0 41.09 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 244μs 793ns 3.07μs 0.494 0 0 41.69 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 312μs 196ns 758ns 6.79 0.453 0 42.91 KB
master RunWaf(args=NestedMap (1000)) net6.0 129μs 64.7ns 242ns 0.453 0 0 34.74 KB
master RunWaf(args=NestedMap (1000)) netcoreapp3.1 159μs 798ns 3.38μs 0.468 0 0 35.39 KB
master RunWaf(args=NestedMap (1000)) net472 216μs 232ns 898ns 5.67 0.428 0 36.33 KB
master RunWafWithAttack(args=Neste(...)tack) [24]) net6.0 210μs 135ns 522ns 0.511 0 0 41.09 KB
master RunWafWithAttack(args=Neste(...)tack) [24]) netcoreapp3.1 245μs 414ns 1.55μs 0.479 0 0 41.7 KB
master RunWafWithAttack(args=Neste(...)tack) [24]) net472 302μs 138ns 534ns 6.81 0.605 0 42.89 KB
#4622 RunWaf(args=NestedMap (10)) net6.0 52.5μs 152ns 591ns 0.212 0 0 16.07 KB
#4622 RunWaf(args=NestedMap (10)) netcoreapp3.1 68.8μs 396ns 3.19μs 0.206 0 0 16.06 KB
#4622 RunWaf(args=NestedMap (10)) net472 98μs 497ns 2.33μs 2.55 0.0944 0 16.14 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 115μs 447ns 1.95μs 0.285 0 0 22.42 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 139μs 718ns 3.66μs 0.265 0 0 22.37 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [22]) net472 172μs 72.2ns 260ns 3.55 0.169 0 22.71 KB
#4622 RunWaf(args=NestedMap (100)) net6.0 128μs 164ns 637ns 0.446 0 0 34.74 KB
#4622 RunWaf(args=NestedMap (100)) netcoreapp3.1 156μs 555ns 2.15μs 0.466 0 0 35.39 KB
#4622 RunWaf(args=NestedMap (100)) net472 215μs 269ns 1.04μs 5.69 0.43 0 36.33 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 195μs 113ns 423ns 0.487 0 0 41.09 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 243μs 1.27μs 5.96μs 0.492 0 0 41.7 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [23]) net472 299μs 210ns 814ns 6.68 0.445 0 42.9 KB
#4622 RunWaf(args=NestedMap (1000)) net6.0 118μs 126ns 486ns 0.473 0 0 34.75 KB
#4622 RunWaf(args=NestedMap (1000)) netcoreapp3.1 164μs 889ns 5.11μs 0.491 0 0 35.39 KB
#4622 RunWaf(args=NestedMap (1000)) net472 215μs 129ns 482ns 5.7 0.43 0 36.34 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [24]) net6.0 201μs 147ns 568ns 0.503 0 0 41.09 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [24]) netcoreapp3.1 247μs 429ns 1.49μs 0.505 0 0 41.69 KB
#4622 RunWafWithAttack(args=Neste(...)tack) [24]) net472 311μs 145ns 543ns 6.67 0.455 0 42.89 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 169μs 183ns 710ns 0.169 0 0 18.09 KB
master SendRequest netcoreapp3.1 191μs 203ns 730ns 0.191 0 0 20.25 KB
master SendRequest net472 0.000273ns 0.000171ns 0.000642ns 0 0 0 0 b
#4622 SendRequest net6.0 168μs 163ns 631ns 0.167 0 0 18.09 KB
#4622 SendRequest netcoreapp3.1 192μs 211ns 815ns 0.192 0 0 20.25 KB
#4622 SendRequest net472 4.25E‑05ns 4.25E‑05ns 0.000165ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4622

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.66 KB 41.94 KB 286 B 0.69%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 512μs 2.53μs 10.4μs 0.519 0 0 41.44 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 642μs 1.41μs 5.47μs 0.319 0 0 41.66 KB
master WriteAndFlushEnrichedTraces net472 787μs 2.56μs 9.59μs 8.44 2.68 0.383 53.24 KB
#4622 WriteAndFlushEnrichedTraces net6.0 492μs 2.17μs 8.41μs 0.498 0 0 41.45 KB
#4622 WriteAndFlushEnrichedTraces netcoreapp3.1 635μs 2.13μs 8.25μs 0.314 0 0 41.94 KB
#4622 WriteAndFlushEnrichedTraces net472 801μs 3.61μs 14μs 8.2 2.34 0.391 53.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.01μs 0.527ns 2.04ns 0.0107 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.33μs 0.702ns 2.72ns 0.0101 0 0 768 B
master ExecuteNonQuery net472 1.6μs 0.822ns 3.08ns 0.116 0 0 730 B
#4622 ExecuteNonQuery net6.0 1.03μs 0.347ns 1.34ns 0.0109 0 0 768 B
#4622 ExecuteNonQuery netcoreapp3.1 1.37μs 0.58ns 2.25ns 0.0103 0 0 768 B
#4622 ExecuteNonQuery net472 1.55μs 1.92ns 6.91ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4622

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync‑net6.0 1.150 1,237.97 1,424.10
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.137 1,171.94 1,332.29

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.17μs 1.53ns 5.94ns 0.014 0 0 992 B
master CallElasticsearch netcoreapp3.1 1.42μs 0.664ns 2.57ns 0.0134 0 0 992 B
master CallElasticsearch net472 2.28μs 0.598ns 2.32ns 0.159 0 0 1 KB
master CallElasticsearchAsync net6.0 1.24μs 0.636ns 2.38ns 0.0137 0 0 968 B
master CallElasticsearchAsync netcoreapp3.1 1.51μs 0.897ns 3.47ns 0.0136 0 0 1.04 KB
master CallElasticsearchAsync net472 2.53μs 0.465ns 1.61ns 0.168 0 0 1.06 KB
#4622 CallElasticsearch net6.0 1.33μs 1.88ns 7.3ns 0.0139 0 0 992 B
#4622 CallElasticsearch netcoreapp3.1 1.39μs 1.26ns 4.71ns 0.0131 0 0 992 B
#4622 CallElasticsearch net472 2.37μs 0.696ns 2.61ns 0.159 0 0 1 KB
#4622 CallElasticsearchAsync net6.0 1.42μs 1.39ns 5.38ns 0.0133 0 0 968 B
#4622 CallElasticsearchAsync netcoreapp3.1 1.55μs 0.612ns 2.37ns 0.0139 0 0 1.04 KB
#4622 CallElasticsearchAsync net472 2.57μs 0.618ns 2.39ns 0.168 0.00129 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.26μs 0.446ns 1.61ns 0.0126 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.52μs 1.52ns 5.69ns 0.0121 0 0 912 B
master ExecuteAsync net472 1.7μs 0.479ns 1.85ns 0.139 0.000847 0 875 B
#4622 ExecuteAsync net6.0 1.3μs 0.599ns 2.32ns 0.013 0 0 912 B
#4622 ExecuteAsync netcoreapp3.1 1.48μs 0.476ns 1.84ns 0.0125 0 0 912 B
#4622 ExecuteAsync net472 1.67μs 0.541ns 1.95ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.77μs 1.71ns 6.64ns 0.0267 0 0 1.94 KB
master SendAsync netcoreapp3.1 4.47μs 1.34ns 5.21ns 0.0336 0 0 2.48 KB
master SendAsync net472 7.11μs 3.09ns 11.6ns 0.482 0 0 3.05 KB
#4622 SendAsync net6.0 3.83μs 1.8ns 6.72ns 0.027 0 0 1.94 KB
#4622 SendAsync netcoreapp3.1 4.46μs 2.5ns 9.36ns 0.0334 0 0 2.48 KB
#4622 SendAsync net472 7.13μs 2.49ns 8.98ns 0.481 0 0 3.05 KB
Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4622

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1 1.142 2,130.98 1,866.77

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.4μs 0.734ns 2.75ns 0.0231 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 2.14μs 8.33ns 32.3ns 0.0222 0 0 1.62 KB
master EnrichedLog net472 2.42μs 0.957ns 3.58ns 0.244 0 0 1.54 KB
#4622 EnrichedLog net6.0 1.46μs 1.67ns 6.47ns 0.0229 0 0 1.62 KB
#4622 EnrichedLog netcoreapp3.1 1.87μs 1.25ns 4.69ns 0.0222 0 0 1.62 KB
#4622 EnrichedLog net472 2.31μs 1.52ns 5.88ns 0.245 0 0 1.54 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 84.8ns 317ns 0.0565 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 206ns 796ns 0 0 0 4.21 KB
master EnrichedLog net472 148μs 154ns 598ns 0.666 0.222 0 4.38 KB
#4622 EnrichedLog net6.0 113μs 87.1ns 326ns 0.0561 0 0 4.21 KB
#4622 EnrichedLog netcoreapp3.1 119μs 87.1ns 337ns 0 0 0 4.21 KB
#4622 EnrichedLog net472 147μs 92.7ns 359ns 0.664 0.221 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.73μs 1.1ns 4.13ns 0.0299 0 0 2.18 KB
master EnrichedLog netcoreapp3.1 3.95μs 1.1ns 4.13ns 0.0297 0 0 2.18 KB
master EnrichedLog net472 4.62μs 1.5ns 5.8ns 0.315 0 0 1.99 KB
#4622 EnrichedLog net6.0 2.88μs 1.1ns 4.1ns 0.0304 0 0 2.18 KB
#4622 EnrichedLog netcoreapp3.1 3.74μs 1.7ns 5.88ns 0.0301 0 0 2.18 KB
#4622 EnrichedLog net472 4.64μs 2.46ns 9.52ns 0.314 0 0 1.99 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.39μs 1.62ns 6.29ns 0.0159 0 0 1.16 KB
master SendReceive netcoreapp3.1 1.61μs 0.653ns 2.44ns 0.016 0 0 1.16 KB
master SendReceive net472 2.04μs 2.17ns 8.12ns 0.184 0 0 1.16 KB
#4622 SendReceive net6.0 1.39μs 2.06ns 8ns 0.016 0 0 1.16 KB
#4622 SendReceive netcoreapp3.1 1.55μs 0.647ns 2.33ns 0.0154 0 0 1.16 KB
#4622 SendReceive net472 2.09μs 0.816ns 2.94ns 0.185 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.42μs 2.35ns 9.1ns 0.0208 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.54μs 1.67ns 6.23ns 0.0212 0 0 1.58 KB
master EnrichedLog net472 4.02μs 1.36ns 5.28ns 0.31 0 0 1.96 KB
#4622 EnrichedLog net6.0 2.51μs 0.625ns 2.34ns 0.0211 0 0 1.53 KB
#4622 EnrichedLog netcoreapp3.1 3.66μs 1.61ns 6.03ns 0.0202 0 0 1.58 KB
#4622 EnrichedLog net472 3.99μs 1.43ns 5.55ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4622

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.141 399.37 455.63
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.124 517.93 582.03

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 399ns 0.126ns 0.473ns 0.00755 0 0 536 B
master StartFinishSpan netcoreapp3.1 518ns 0.117ns 0.438ns 0.00717 0 0 536 B
master StartFinishSpan net472 612ns 0.452ns 1.75ns 0.0853 0 0 538 B
master StartFinishScope net6.0 509ns 0.802ns 3ns 0.00908 0 0 656 B
master StartFinishScope netcoreapp3.1 757ns 0.267ns 0.962ns 0.00907 0 0 656 B
master StartFinishScope net472 805ns 0.916ns 3.55ns 0.0982 0 0 618 B
#4622 StartFinishSpan net6.0 456ns 0.223ns 0.803ns 0.00758 0 0 536 B
#4622 StartFinishSpan netcoreapp3.1 585ns 1.85ns 7.17ns 0.00728 0 0 536 B
#4622 StartFinishSpan net472 633ns 0.369ns 1.43ns 0.0851 0 0 538 B
#4622 StartFinishScope net6.0 522ns 0.174ns 0.652ns 0.00915 0 0 656 B
#4622 StartFinishScope netcoreapp3.1 698ns 1.89ns 7.3ns 0.00875 0 0 656 B
#4622 StartFinishScope net472 811ns 0.263ns 0.983ns 0.0981 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 583ns 0.222ns 0.861ns 0.00929 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 805ns 0.4ns 1.55ns 0.00879 0 0 656 B
master RunOnMethodBegin net472 958ns 0.229ns 0.856ns 0.0977 0 0 618 B
#4622 RunOnMethodBegin net6.0 563ns 0.229ns 0.887ns 0.00923 0 0 656 B
#4622 RunOnMethodBegin netcoreapp3.1 846ns 0.362ns 1.36ns 0.00903 0 0 656 B
#4622 RunOnMethodBegin net472 930ns 0.834ns 3.23ns 0.0981 0 0 618 B

@andrewlock
Copy link
Member Author

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4622) (11.688M)   : 0, 11688288
    master (11.668M)   : 0, 11668341
    benchmarks/2.37.0 (11.517M)   : 0, 11516696
    benchmarks/2.9.0 (11.497M)   : 0, 11496834

    section Automatic
    This PR (4622) (8.271M)   : 0, 8270797
    master (8.148M)   : 0, 8147775
    benchmarks/2.37.0 (8.126M)   : 0, 8125502
    benchmarks/2.9.0 (8.363M)   : 0, 8363413

    section Trace stats
    master (8.123M)   : 0, 8123030
    benchmarks/2.37.0 (8.166M)   : 0, 8165926

    section Manual
    This PR (4622) (10.288M)   : 0, 10288489
    master (10.315M)   : 0, 10315013
    benchmarks/2.37.0 (10.292M)   : 0, 10292442

    section Manual + Automatic
    This PR (4622) (7.786M)   : 0, 7786414
    master (7.852M)   : 0, 7851922
    benchmarks/2.37.0 (7.731M)   : 0, 7731254

    section Version Conflict
    master (7.073M)   : 0, 7072639
    benchmarks/2.37.0 (6.879M)   : 0, 6878658
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4622) (9.609M)   : 0, 9608878
    master (9.674M)   : 0, 9674353
    benchmarks/2.37.0 (9.258M)   : 0, 9257590
    benchmarks/2.9.0 (9.458M)   : 0, 9457529

    section Automatic
    This PR (4622) (6.767M)   : 0, 6767466
    master (6.682M)   : 0, 6682295
    benchmarks/2.37.0 (6.795M)   : 0, 6795420

    section Trace stats
    master (6.854M)   : 0, 6854008
    benchmarks/2.37.0 (6.555M)   : 0, 6555408

    section Manual
    This PR (4622) (8.626M)   : 0, 8625865
    master (8.489M)   : 0, 8488738
    benchmarks/2.37.0 (8.512M)   : 0, 8512310

    section Manual + Automatic
    This PR (4622) (6.421M)   : 0, 6420526
    master (6.650M)   : 0, 6650081
    benchmarks/2.37.0 (6.489M)   : 0, 6488876

    section Version Conflict
    master (5.999M)   : 0, 5998650
    benchmarks/2.37.0 (5.888M)   : 0, 5887538
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4622) (9.619M)   : 0, 9618896
    master (9.438M)   : 0, 9438307
    benchmarks/2.37.0 (9.257M)   : 0, 9257263
    benchmarks/2.9.0 (9.321M)   : 0, 9320552

    section Automatic
    This PR (4622) (6.723M)   : 0, 6723035
    master (6.786M)   : 0, 6786097
    benchmarks/2.37.0 (6.645M)   : 0, 6644825
    benchmarks/2.9.0 (6.803M)   : 0, 6803324

    section Trace stats
    master (6.764M)   : 0, 6764366
    benchmarks/2.37.0 (6.627M)   : 0, 6626639

    section Manual
    This PR (4622) (8.400M)   : 0, 8400126
    master (8.603M)   : 0, 8602779
    benchmarks/2.37.0 (8.211M)   : 0, 8211150

    section Manual + Automatic
    This PR (4622) (6.455M)   : 0, 6454879
    master (6.473M)   : 0, 6472719
    benchmarks/2.37.0 (6.448M)   : 0, 6447797

    section Version Conflict
    master (6.067M)   : 0, 6067268
    benchmarks/2.37.0 (6.020M)   : 0, 6019736
Loading 
gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4622) (7.380M)   : 0, 7380159
    master (7.194M)   : 0, 7193639
    benchmarks/2.37.0 (7.436M)   : 0, 7435584
    benchmarks/2.9.0 (7.452M)   : 0, 7452008

    section No attack
    This PR (4622) (2.069M)   : 0, 2069196
    master (2.018M)   : 0, 2017534
    benchmarks/2.37.0 (2.086M)   : 0, 2085601
    benchmarks/2.9.0 (3.140M)   : 0, 3139837

    section Attack
    This PR (4622) (1.648M)   : 0, 1647842
    master (1.634M)   : 0, 1633950
    benchmarks/2.37.0 (1.629M)   : 0, 1629220
    benchmarks/2.9.0 (2.461M)   : 0, 2460581

    section Blocking
    This PR (4622) (3.189M)   : 0, 3189246
    master (3.161M)   : 0, 3160582
    benchmarks/2.37.0 (2.988M)   : 0, 2988404
Loading

@andrewlock andrewlock merged commit 8d07286 into master Sep 14, 2023
51 checks passed
@andrewlock andrewlock deleted the andrew/ci/flaky-metrics branch September 14, 2023 14:47
@github-actions github-actions bot added this to the vNext milestone Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tonyredondo tonyredondo tonyredondo approved these changes

@kevingosse kevingosse kevingosse approved these changes

Assignees
No one assigned
Labels
area:tests unit tests, integration tests
Projects
None yet
Milestone
2.38.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@andrewlock @tonyredondo @kevingosse