Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASM] Skip flaky test #6771

Closed
wants to merge 1 commit into from
Closed

[ASM] Skip flaky test #6771

wants to merge 1 commit into from
+1 −1

Conversation

andrewlock
Copy link
Member

Summary of changes

Skips the flaky gRPC IAST test

Reason for change

The test has been failing since 10th March

Implementation details

The snapshot is often missing the metastruct

Test coverage

Less after this - we should investigate this ASAP

@andrewlock andrewlock added the area:tests unit tests, integration tests label Mar 17, 2025
@andrewlock andrewlock requested a review from a team as a code owner March 17, 2025 09:51
@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Mar 17, 2025 •
edited
Loading

Datadog Report

Branch report: andrew/ci/skip-grpc-tests
Commit report: 2c737d3
Test service: dd-trace-dotnet

✅ 0 Failed, 252024 Passed, 2417 Skipped, 20h 7m 5.06s Total Time

@andrewlock
Copy link
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (69ms)  : 66, 72
     .   : milestone, 69,
    master - mean (69ms)  : 67, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (1,003ms)  : 980, 1025
     .   : milestone, 1003,
    master - mean (1,005ms)  : 983, 1027
     .   : milestone, 1005,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (102ms)  : 100, 104
     .   : milestone, 102,
    master - mean (103ms)  : 101, 105
     .   : milestone, 103,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (683ms)  : 667, 700
     .   : milestone, 683,
    master - mean (687ms)  : 668, 707
     .   : milestone, 687,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (90ms)  : 88, 91
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (638ms)  : 619, 658
     .   : milestone, 638,
    master - mean (644ms)  : 624, 664
     .   : milestone, 644,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (190ms)  : 186, 194
     .   : milestone, 190,
    master - mean (190ms)  : 187, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (1,104ms)  : 1078, 1130
     .   : milestone, 1104,
    master - mean (1,106ms)  : 1080, 1133
     .   : milestone, 1106,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (269ms)  : 265, 273
     .   : milestone, 269,
    master - mean (270ms)  : 265, 274
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (874ms)  : 841, 906
     .   : milestone, 874,
    master - mean (874ms)  : 851, 897
     .   : milestone, 874,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6771) - mean (262ms)  : 257, 266
     .   : milestone, 262,
    master - mean (262ms)  : 259, 266
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6771) - mean (854ms)  : 815, 893
     .   : milestone, 854,
    master - mean (856ms)  : 825, 887
     .   : milestone, 856,

@andrewlock
Copy link
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #6771 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.121
  • 2 benchmarks are slower, with geometric mean 1.245
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.02μs 43.7ns 255ns 0.0152 0.0076 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10.1μs 53.5ns 268ns 0.0185 0.00926 0 5.8 KB
master StartStopWithChild net472 15.9μs 34.8ns 125ns 1.05 0.31 0.103 6.21 KB
#6771 StartStopWithChild net6.0 7.94μs 41.8ns 248ns 0.0163 0.00815 0 5.61 KB
#6771 StartStopWithChild netcoreapp3.1 10.2μs 55.1ns 326ns 0.0255 0.0102 0 5.8 KB
#6771 StartStopWithChild net472 16.2μs 47.7ns 185ns 1.03 0.292 0.0867 6.22 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 513μs 639ns 2.47μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 648μs 632ns 2.45μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 865μs 901ns 3.49μs 0.431 0 0 3.3 KB
#6771 WriteAndFlushEnrichedTraces net6.0 503μs 301ns 1.09μs 0 0 0 2.7 KB
#6771 WriteAndFlushEnrichedTraces netcoreapp3.1 663μs 613ns 2.37μs 0 0 0 2.7 KB
#6771 WriteAndFlushEnrichedTraces net472 851μs 787ns 2.94μs 0.422 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 132μs 527ns 2.04μs 0.133 0 0 14.47 KB
master SendRequest netcoreapp3.1 150μs 329ns 1.27μs 0.15 0 0 17.27 KB
master SendRequest net472 0.000368ns 0.000142ns 0.000548ns 0 0 0 0 b
#6771 SendRequest net6.0 132μs 346ns 1.34μs 0.195 0 0 14.47 KB
#6771 SendRequest netcoreapp3.1 149μs 283ns 1.1μs 0.149 0 0 17.27 KB
#6771 SendRequest net472 0.00112ns 0.000526ns 0.00197ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 592μs 3.4μs 28.8μs 0.541 0 0 41.6 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 726μs 4.08μs 37.4μs 0.353 0 0 41.77 KB
master WriteAndFlushEnrichedTraces net472 872μs 4.34μs 19.9μs 8.08 2.55 0.425 53.28 KB
#6771 WriteAndFlushEnrichedTraces net6.0 602μs 3.48μs 29.7μs 0.556 0 0 41.67 KB
#6771 WriteAndFlushEnrichedTraces netcoreapp3.1 676μs 3.2μs 16.9μs 0.329 0 0 41.8 KB
#6771 WriteAndFlushEnrichedTraces net472 873μs 3.14μs 12.2μs 8.25 2.6 0.434 53.3 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.34μs 1.46ns 5.66ns 0.0145 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.85μs 2.48ns 9.6ns 0.0131 0 0 1.02 KB
master ExecuteNonQuery net472 2.03μs 1.02ns 3.66ns 0.156 0.00101 0 987 B
#6771 ExecuteNonQuery net6.0 1.39μs 1.85ns 7.17ns 0.0146 0 0 1.02 KB
#6771 ExecuteNonQuery netcoreapp3.1 1.86μs 2.98ns 11.6ns 0.014 0 0 1.02 KB
#6771 ExecuteNonQuery net472 2.09μs 1.84ns 6.87ns 0.156 0.00104 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.18μs 0.505ns 1.89ns 0.0135 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.62μs 1.83ns 6.35ns 0.0132 0 0 976 B
master CallElasticsearch net472 2.57μs 1.65ns 6.4ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.31μs 0.742ns 2.78ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.66μs 0.987ns 3.69ns 0.0137 0 0 1.02 KB
master CallElasticsearchAsync net472 2.71μs 0.944ns 3.53ns 0.166 0 0 1.05 KB
#6771 CallElasticsearch net6.0 1.28μs 0.792ns 3.07ns 0.0135 0 0 976 B
#6771 CallElasticsearch netcoreapp3.1 1.56μs 0.794ns 3.07ns 0.0131 0 0 976 B
#6771 CallElasticsearch net472 2.69μs 1.58ns 6.12ns 0.157 0 0 995 B
#6771 CallElasticsearchAsync net6.0 1.34μs 0.862ns 3.34ns 0.0134 0 0 952 B
#6771 CallElasticsearchAsync netcoreapp3.1 1.71μs 0.898ns 3.48ns 0.0136 0 0 1.02 KB
#6771 CallElasticsearchAsync net472 2.72μs 2.08ns 8.06ns 0.167 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.39μs 0.442ns 1.71ns 0.0135 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.62μs 0.508ns 1.83ns 0.013 0 0 952 B
master ExecuteAsync net472 1.85μs 0.519ns 2.01ns 0.144 0 0 915 B
#6771 ExecuteAsync net6.0 1.38μs 1.57ns 6.09ns 0.0131 0 0 952 B
#6771 ExecuteAsync netcoreapp3.1 1.7μs 1.04ns 4.03ns 0.0127 0 0 952 B
#6771 ExecuteAsync net472 1.77μs 0.815ns 3.16ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.31μs 2.33ns 8.41ns 0.0324 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.51μs 2.65ns 10.2ns 0.0385 0 0 2.85 KB
master SendAsync net472 7.47μs 2.4ns 8.97ns 0.493 0 0 3.12 KB
#6771 SendAsync net6.0 4.35μs 2.76ns 10.3ns 0.0306 0 0 2.31 KB
#6771 SendAsync netcoreapp3.1 5.28μs 3.55ns 13.7ns 0.0368 0 0 2.85 KB
#6771 SendAsync net472 7.54μs 2.08ns 7.49ns 0.494 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.52μs 0.618ns 2.23ns 0.0228 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.11μs 0.772ns 2.78ns 0.0222 0 0 1.64 KB
master EnrichedLog net472 2.47μs 0.796ns 3.08ns 0.249 0 0 1.57 KB
#6771 EnrichedLog net6.0 1.44μs 1.11ns 4.28ns 0.023 0 0 1.64 KB
#6771 EnrichedLog netcoreapp3.1 2.24μs 0.862ns 3.11ns 0.0223 0 0 1.64 KB
#6771 EnrichedLog net472 2.49μs 0.607ns 2.35ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 111μs 126ns 471ns 0.0555 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 117μs 243ns 941ns 0.0578 0 0 4.28 KB
master EnrichedLog net472 149μs 149ns 578ns 0.671 0.224 0 4.46 KB
#6771 EnrichedLog net6.0 111μs 151ns 585ns 0.0554 0 0 4.28 KB
#6771 EnrichedLog netcoreapp3.1 115μs 219ns 847ns 0.0573 0 0 4.28 KB
#6771 EnrichedLog net472 148μs 136ns 526ns 0.668 0.223 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.98μs 0.767ns 2.97ns 0.0314 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.33μs 1.7ns 6.58ns 0.0282 0 0 2.2 KB
master EnrichedLog net472 4.75μs 1.28ns 4.79ns 0.319 0 0 2.02 KB
#6771 EnrichedLog net6.0 3.14μs 0.711ns 2.76ns 0.0299 0 0 2.2 KB
#6771 EnrichedLog netcoreapp3.1 4.2μs 1.88ns 7.29ns 0.0287 0 0 2.2 KB
#6771 EnrichedLog net472 4.84μs 0.833ns 3.23ns 0.321 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 0.981ns 3.8ns 0.0163 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.75μs 1.07ns 4.16ns 0.0158 0 0 1.14 KB
master SendReceive net472 2.17μs 2.23ns 8.65ns 0.183 0 0 1.16 KB
#6771 SendReceive net6.0 1.42μs 1.42ns 5.49ns 0.0163 0 0 1.14 KB
#6771 SendReceive netcoreapp3.1 1.67μs 0.674ns 2.52ns 0.0151 0 0 1.14 KB
#6771 SendReceive net472 2.08μs 0.977ns 3.79ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.75μs 0.676ns 2.62ns 0.022 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 4.02μs 1.56ns 6.05ns 0.0221 0 0 1.65 KB
master EnrichedLog net472 4.25μs 2.35ns 9.1ns 0.322 0 0 2.04 KB
#6771 EnrichedLog net6.0 2.76μs 1.28ns 4.97ns 0.0221 0 0 1.6 KB
#6771 EnrichedLog netcoreapp3.1 3.83μs 1.36ns 5.25ns 0.0213 0 0 1.65 KB
#6771 EnrichedLog net472 4.23μs 2.22ns 8.31ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6771

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.284 486.70 625.00
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.208 558.17 674.01

Faster 🎉 in #6771

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.121 470.08 419.38

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 470ns 0.0811ns 0.314ns 0.008 0 0 576 B
master StartFinishSpan netcoreapp3.1 558ns 0.324ns 1.26ns 0.00777 0 0 576 B
master StartFinishSpan net472 649ns 0.193ns 0.747ns 0.0917 0 0 578 B
master StartFinishScope net6.0 487ns 0.119ns 0.446ns 0.0098 0 0 696 B
master StartFinishScope netcoreapp3.1 795ns 0.799ns 3.1ns 0.0091 0 0 696 B
master StartFinishScope net472 781ns 0.377ns 1.46ns 0.104 0 0 658 B
#6771 StartFinishSpan net6.0 419ns 0.142ns 0.55ns 0.00807 0 0 576 B
#6771 StartFinishSpan netcoreapp3.1 673ns 1.04ns 4.02ns 0.00773 0 0 576 B
#6771 StartFinishSpan net472 595ns 0.223ns 0.835ns 0.0918 0 0 578 B
#6771 StartFinishScope net6.0 625ns 0.585ns 2.27ns 0.00974 0 0 696 B
#6771 StartFinishScope netcoreapp3.1 747ns 4.02ns 21.6ns 0.00929 0 0 696 B
#6771 StartFinishScope net472 865ns 0.444ns 1.72ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 640ns 0.206ns 0.798ns 0.00961 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 899ns 0.407ns 1.41ns 0.00932 0 0 696 B
master RunOnMethodBegin net472 1.17μs 0.405ns 1.57ns 0.104 0 0 658 B
#6771 RunOnMethodBegin net6.0 586ns 0.0912ns 0.329ns 0.00966 0 0 696 B
#6771 RunOnMethodBegin netcoreapp3.1 999ns 0.983ns 3.81ns 0.0094 0 0 696 B
#6771 RunOnMethodBegin net472 1.21μs 0.336ns 1.26ns 0.104 0 0 658 B

@andrewlock
Copy link
Member Author

Benchmarks Report for appsec 🐌

Benchmarks for #6771 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.182
  • 2 benchmarks are slower, with geometric mean 1.119
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6771

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0 1.122 195,643.81 219,568.33
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0 1.115 190,066.12 211,886.65

Faster 🎉 in #6771

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472 1.182 4,321.95 3,657.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 190μs 72.1ns 270ns 2.75 0 0 196.14 KB
master AllCycleSimpleBody netcoreapp3.1 303μs 217ns 810ns 2.69 0 0 203.7 KB
master AllCycleSimpleBody net472 264μs 71.1ns 275ns 37.2 2.11 0 234.72 KB
master AllCycleMoreComplexBody net6.0 196μs 125ns 484ns 2.84 0 0 199.64 KB
master AllCycleMoreComplexBody netcoreapp3.1 302μs 155ns 558ns 2.72 0 0 207.12 KB
master AllCycleMoreComplexBody net472 276μs 137ns 492ns 37.7 2.21 0 238.24 KB
master ObjectExtractorSimpleBody net6.0 136ns 0.0659ns 0.238ns 0.00391 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 199ns 0.2ns 0.777ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 165ns 0.0976ns 0.365ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.84μs 2.53ns 9.78ns 0.0526 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.68μs 3.33ns 12.9ns 0.0494 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.32μs 2.68ns 10.4ns 0.602 0.00649 0 3.8 KB
#6771 AllCycleSimpleBody net6.0 212μs 83.3ns 312ns 2.75 0 0 195.99 KB
#6771 AllCycleSimpleBody netcoreapp3.1 320μs 107ns 399ns 2.69 0 0 203.56 KB
#6771 AllCycleSimpleBody net472 283μs 131ns 509ns 37.2 2.14 0 234.56 KB
#6771 AllCycleMoreComplexBody net6.0 220μs 85.8ns 309ns 2.74 0 0 199.5 KB
#6771 AllCycleMoreComplexBody netcoreapp3.1 324μs 104ns 388ns 2.75 0 0 206.98 KB
#6771 AllCycleMoreComplexBody net472 289μs 139ns 519ns 37.8 2.17 0 238.08 KB
#6771 ObjectExtractorSimpleBody net6.0 139ns 0.149ns 0.578ns 0.00393 0 0 280 B
#6771 ObjectExtractorSimpleBody netcoreapp3.1 191ns 0.201ns 0.778ns 0.00376 0 0 272 B
#6771 ObjectExtractorSimpleBody net472 166ns 0.143ns 0.552ns 0.0445 0 0 281 B
#6771 ObjectExtractorMoreComplexBody net6.0 3.02μs 1.99ns 7.69ns 0.0527 0 0 3.78 KB
#6771 ObjectExtractorMoreComplexBody netcoreapp3.1 3.89μs 2.99ns 11.6ns 0.0503 0 0 3.69 KB
#6771 ObjectExtractorMoreComplexBody net472 3.66μs 2.6ns 10.1ns 0.602 0.00549 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 36.9μs 61.1ns 229ns 0.457 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.1μs 17.7ns 68.7ns 0.432 0 0 32.4 KB
master EncodeArgs net472 64.9μs 36.8ns 138ns 5.14 0.0646 0 32.5 KB
master EncodeLegacyArgs net6.0 79.2μs 433ns 2.63μs 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 111μs 359ns 1.39μs 0 0 0 2.14 KB
master EncodeLegacyArgs net472 151μs 81.7ns 316ns 0.304 0 0 2.15 KB
#6771 EncodeArgs net6.0 37μs 19.7ns 76.1ns 0.463 0 0 32.4 KB
#6771 EncodeArgs netcoreapp3.1 54.5μs 16.8ns 63ns 0.438 0 0 32.4 KB
#6771 EncodeArgs net472 65.7μs 42.7ns 165ns 5.14 0.0659 0 32.5 KB
#6771 EncodeLegacyArgs net6.0 77.5μs 33.9ns 127ns 0 0 0 2.14 KB
#6771 EncodeLegacyArgs netcoreapp3.1 107μs 431ns 1.67μs 0 0 0 2.14 KB
#6771 EncodeLegacyArgs net472 151μs 193ns 748ns 0.305 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 173μs 77.5ns 280ns 0 0 0 2.54 KB
master RunWafRealisticBenchmark netcoreapp3.1 188μs 205ns 766ns 0 0 0 2.49 KB
master RunWafRealisticBenchmark net472 201μs 484ns 1.88μs 0.405 0 0 2.55 KB
master RunWafRealisticBenchmarkWithAttack net6.0 116μs 64.8ns 251ns 0 0 0 1.57 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 124μs 357ns 1.38μs 0 0 0 1.55 KB
master RunWafRealisticBenchmarkWithAttack net472 132μs 115ns 447ns 0.197 0 0 1.58 KB
#6771 RunWafRealisticBenchmark net6.0 180μs 81.1ns 292ns 0 0 0 2.54 KB
#6771 RunWafRealisticBenchmark netcoreapp3.1 185μs 221ns 856ns 0 0 0 2.49 KB
#6771 RunWafRealisticBenchmark net472 200μs 59.2ns 229ns 0.399 0 0 2.55 KB
#6771 RunWafRealisticBenchmarkWithAttack net6.0 115μs 92ns 356ns 0 0 0 1.57 KB
#6771 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 122μs 38.1ns 148ns 0 0 0 1.55 KB
#6771 RunWafRealisticBenchmarkWithAttack net472 132μs 110ns 426ns 0.197 0 0 1.58 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6771

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 59.07 KB 60.49 KB 1.42 KB 2.40%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 60.7μs 986ns 9.86μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 61.1μs 742ns 7.38μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 36.9μs 172ns 643ns 0 0 0 59.07 KB
master StringConcatAspectBenchmark net6.0 301μs 5.75μs 56.9μs 0 0 0 254.9 KB
master StringConcatAspectBenchmark netcoreapp3.1 338μs 1.87μs 11.4μs 0 0 0 252.49 KB
master StringConcatAspectBenchmark net472 293μs 7.07μs 69.6μs 0 0 0 278.53 KB
#6771 StringConcatBenchmark net6.0 56.3μs 578ns 5.78μs 0 0 0 43.44 KB
#6771 StringConcatBenchmark netcoreapp3.1 61.8μs 873ns 8.55μs 0 0 0 42.64 KB
#6771 StringConcatBenchmark net472 36.6μs 83.2ns 300ns 0 0 0 60.49 KB
#6771 StringConcatAspectBenchmark net6.0 320μs 1.8μs 12.2μs 0 0 0 254.65 KB
#6771 StringConcatAspectBenchmark netcoreapp3.1 337μs 1.78μs 12.7μs 0 0 0 253.29 KB
#6771 StringConcatAspectBenchmark net472 280μs 6.31μs 60.9μs 0 0 0 278.53 KB

@andrewlock
Copy link
Member Author

andrewlock commented Mar 17, 2025 •
edited
Loading

Superseded (because it's fixed) by

@andrewlock andrewlock closed this Mar 17, 2025
@andrewlock andrewlock deleted the andrew/ci/skip-grpc-tests branch March 17, 2025 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@e-n-0 e-n-0

Assignees
No one assigned
Labels
area:asm area:tests unit tests, integration tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @e-n-0