Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove graphql scenarios from CI system tests #6768

Merged
merged 1 commit into from
Mar 14, 2025
Merged

Remove graphql scenarios from CI system tests #6768

merged 1 commit into from
Mar 14, 2025
+1 −4

Conversation

e-n-0
Copy link
Member

@e-n-0 e-n-0 commented Mar 14, 2025

Summary of changes

Remove the graphql_scenarios key from the script that loads the system tests in the CI because it was removed from the system tests repo.

@e-n-0 e-n-0 requested a review from a team as a code owner March 14, 2025 12:26
@github-actions github-actions bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Mar 14, 2025
@andrewlock
Copy link
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (69ms)  : 65, 73
     .   : milestone, 69,
    master - mean (69ms)  : 66, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (1,002ms)  : 981, 1023
     .   : milestone, 1002,
    master - mean (1,002ms)  : 982, 1022
     .   : milestone, 1002,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (102ms)  : 100, 104
     .   : milestone, 102,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (685ms)  : 670, 700
     .   : milestone, 685,
    master - mean (685ms)  : 667, 703
     .   : milestone, 685,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 86, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (641ms)  : 622, 659
     .   : milestone, 641,
    master - mean (643ms)  : 623, 663
     .   : milestone, 643,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (189ms)  : 186, 193
     .   : milestone, 189,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (1,103ms)  : 1077, 1129
     .   : milestone, 1103,
    master - mean (1,102ms)  : 1073, 1131
     .   : milestone, 1102,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (268ms)  : 265, 272
     .   : milestone, 268,
    master - mean (269ms)  : 263, 275
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (875ms)  : 847, 904
     .   : milestone, 875,
    master - mean (874ms)  : 836, 912
     .   : milestone, 874,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6768) - mean (262ms)  : 258, 267
     .   : milestone, 262,
    master - mean (262ms)  : 256, 267
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6768) - mean (853ms)  : 827, 879
     .   : milestone, 853,
    master - mean (858ms)  : 824, 892
     .   : milestone, 858,

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Mar 14, 2025 •
edited
Loading

Datadog Report

Branch report: flavien/fix-system-tests-ci
Commit report: 7f06b2b
Test service: dd-trace-dotnet

✅ 0 Failed, 253817 Passed, 2403 Skipped, 19h 34m 21.06s Total Time

@andrewlock
Copy link
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6768 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.175
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.08μs 45.8ns 318ns 0.0116 0.00387 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10.1μs 54ns 275ns 0.0147 0.0049 0 5.8 KB
master StartStopWithChild net472 16μs 32.4ns 125ns 1.04 0.302 0.0953 6.21 KB
#6768 StartStopWithChild net6.0 8.08μs 42.3ns 220ns 0.0121 0.00402 0 5.61 KB
#6768 StartStopWithChild netcoreapp3.1 9.87μs 48.7ns 229ns 0.0239 0.00957 0 5.8 KB
#6768 StartStopWithChild net472 16.1μs 58.8ns 228ns 1.05 0.309 0.106 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 502μs 532ns 2.06μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 651μs 706ns 2.73μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 845μs 877ns 3.4μs 0.422 0 0 3.3 KB
#6768 WriteAndFlushEnrichedTraces net6.0 511μs 1.68μs 6.06μs 0 0 0 2.7 KB
#6768 WriteAndFlushEnrichedTraces netcoreapp3.1 642μs 312ns 1.17μs 0 0 0 2.7 KB
#6768 WriteAndFlushEnrichedTraces net472 870μs 514ns 1.92μs 0.434 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 135μs 287ns 1.07μs 0.134 0 0 14.47 KB
master SendRequest netcoreapp3.1 149μs 258ns 997ns 0.223 0 0 17.27 KB
master SendRequest net472 0.00108ns 0.000391ns 0.00146ns 0 0 0 0 b
#6768 SendRequest net6.0 132μs 413ns 1.6μs 0.192 0 0 14.47 KB
#6768 SendRequest netcoreapp3.1 151μs 138ns 535ns 0.224 0 0 17.27 KB
#6768 SendRequest net472 0.0016ns 0.000518ns 0.00201ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 567μs 2.67μs 10.3μs 0.566 0 0 41.66 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 693μs 3.73μs 21.7μs 0.361 0 0 41.7 KB
master WriteAndFlushEnrichedTraces net472 845μs 3.86μs 14.9μs 8.81 2.52 0.419 53.31 KB
#6768 WriteAndFlushEnrichedTraces net6.0 571μs 3.11μs 20.7μs 0.573 0 0 41.53 KB
#6768 WriteAndFlushEnrichedTraces netcoreapp3.1 689μs 3.57μs 16.7μs 0.329 0 0 41.64 KB
#6768 WriteAndFlushEnrichedTraces net472 889μs 4.34μs 17.4μs 8.04 2.23 0.446 53.34 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.41μs 2.29ns 8.88ns 0.0139 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.8μs 1.48ns 5.52ns 0.0135 0 0 1.02 KB
master ExecuteNonQuery net472 2.2μs 2.86ns 11.1ns 0.156 0.00109 0 987 B
#6768 ExecuteNonQuery net6.0 1.36μs 0.971ns 3.63ns 0.0143 0 0 1.02 KB
#6768 ExecuteNonQuery netcoreapp3.1 1.71μs 1.21ns 4.69ns 0.0137 0 0 1.02 KB
#6768 ExecuteNonQuery net472 2.14μs 3.24ns 11.7ns 0.157 0.00107 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.25μs 1.05ns 3.77ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.58μs 1.17ns 4.51ns 0.0125 0 0 976 B
master CallElasticsearch net472 2.62μs 2.15ns 8.32ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.32μs 0.874ns 3.39ns 0.0133 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.7μs 0.724ns 2.8ns 0.0136 0 0 1.02 KB
master CallElasticsearchAsync net472 2.64μs 3.54ns 13.7ns 0.166 0 0 1.05 KB
#6768 CallElasticsearch net6.0 1.22μs 0.319ns 1.19ns 0.0134 0 0 976 B
#6768 CallElasticsearch netcoreapp3.1 1.57μs 2.12ns 8.21ns 0.0134 0 0 976 B
#6768 CallElasticsearch net472 2.59μs 0.876ns 3.28ns 0.157 0 0 995 B
#6768 CallElasticsearchAsync net6.0 1.32μs 0.625ns 2.42ns 0.0132 0 0 952 B
#6768 CallElasticsearchAsync netcoreapp3.1 1.71μs 1.17ns 4.37ns 0.0137 0 0 1.02 KB
#6768 CallElasticsearchAsync net472 2.7μs 4.92ns 18.4ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.36μs 3.04ns 11.8ns 0.0136 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.66μs 0.994ns 3.72ns 0.0124 0 0 952 B
master ExecuteAsync net472 1.81μs 0.574ns 2.15ns 0.145 0 0 915 B
#6768 ExecuteAsync net6.0 1.24μs 1.04ns 3.88ns 0.013 0 0 952 B
#6768 ExecuteAsync netcoreapp3.1 1.64μs 0.702ns 2.53ns 0.0124 0 0 952 B
#6768 ExecuteAsync net472 1.8μs 0.363ns 1.31ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.35μs 1.86ns 6.95ns 0.0325 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.5μs 1.92ns 7.44ns 0.0384 0 0 2.85 KB
master SendAsync net472 7.48μs 1.81ns 7ns 0.494 0 0 3.12 KB
#6768 SendAsync net6.0 4.31μs 1.74ns 6.75ns 0.0323 0 0 2.31 KB
#6768 SendAsync netcoreapp3.1 5.51μs 2.2ns 8.53ns 0.0385 0 0 2.85 KB
#6768 SendAsync net472 7.43μs 1.5ns 5.79ns 0.495 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6768

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1 1.175 2,423.58 2,062.14

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.5μs 0.728ns 2.73ns 0.0229 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.42μs 1.2ns 4.63ns 0.0218 0 0 1.64 KB
master EnrichedLog net472 2.42μs 0.772ns 2.99ns 0.249 0 0 1.57 KB
#6768 EnrichedLog net6.0 1.52μs 0.421ns 1.52ns 0.0234 0 0 1.64 KB
#6768 EnrichedLog netcoreapp3.1 2.06μs 1.63ns 5.66ns 0.0224 0 0 1.64 KB
#6768 EnrichedLog net472 2.66μs 0.86ns 3.33ns 0.25 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 111μs 158ns 593ns 0.055 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 117μs 198ns 767ns 0 0 0 4.28 KB
master EnrichedLog net472 149μs 153ns 591ns 0.67 0.223 0 4.46 KB
#6768 EnrichedLog net6.0 112μs 134ns 519ns 0.0555 0 0 4.28 KB
#6768 EnrichedLog netcoreapp3.1 115μs 203ns 761ns 0.057 0 0 4.28 KB
#6768 EnrichedLog net472 149μs 124ns 465ns 0.668 0.223 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3μs 0.605ns 2.26ns 0.03 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.29μs 3.24ns 12.5ns 0.03 0 0 2.2 KB
master EnrichedLog net472 4.83μs 1.47ns 5.71ns 0.321 0 0 2.02 KB
#6768 EnrichedLog net6.0 2.98μs 0.577ns 2.08ns 0.0297 0 0 2.2 KB
#6768 EnrichedLog netcoreapp3.1 4.16μs 2.82ns 10.9ns 0.029 0 0 2.2 KB
#6768 EnrichedLog net472 4.69μs 1.19ns 4.27ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 0.731ns 2.83ns 0.0156 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.79μs 1.35ns 5.23ns 0.0152 0 0 1.14 KB
master SendReceive net472 2.09μs 1ns 3.88ns 0.183 0 0 1.16 KB
#6768 SendReceive net6.0 1.43μs 1.54ns 5.98ns 0.0159 0 0 1.14 KB
#6768 SendReceive netcoreapp3.1 1.7μs 0.603ns 2.34ns 0.0153 0 0 1.14 KB
#6768 SendReceive net472 2.1μs 0.5ns 1.87ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.77μs 1.32ns 4.95ns 0.0222 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.89μs 4.18ns 16.2ns 0.0213 0 0 1.65 KB
master EnrichedLog net472 4.43μs 1.91ns 7.4ns 0.323 0 0 2.04 KB
#6768 EnrichedLog net6.0 2.82μs 0.735ns 2.85ns 0.0226 0 0 1.6 KB
#6768 EnrichedLog netcoreapp3.1 3.89μs 1.93ns 7.49ns 0.0214 0 0 1.65 KB
#6768 EnrichedLog net472 4.33μs 2.07ns 8.01ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 455ns 0.131ns 0.506ns 0.00803 0 0 576 B
master StartFinishSpan netcoreapp3.1 595ns 0.381ns 1.48ns 0.00772 0 0 576 B
master StartFinishSpan net472 688ns 1.76ns 6.8ns 0.0918 0 0 578 B
master StartFinishScope net6.0 484ns 0.0714ns 0.277ns 0.00988 0 0 696 B
master StartFinishScope netcoreapp3.1 776ns 0.455ns 1.76ns 0.00936 0 0 696 B
master StartFinishScope net472 850ns 0.291ns 1.13ns 0.104 0 0 658 B
#6768 StartFinishSpan net6.0 427ns 0.167ns 0.646ns 0.00809 0 0 576 B
#6768 StartFinishSpan netcoreapp3.1 633ns 0.279ns 1.08ns 0.0078 0 0 576 B
#6768 StartFinishSpan net472 742ns 0.175ns 0.679ns 0.0915 0 0 578 B
#6768 StartFinishScope net6.0 504ns 0.107ns 0.416ns 0.00985 0 0 696 B
#6768 StartFinishScope netcoreapp3.1 745ns 0.598ns 2.32ns 0.00935 0 0 696 B
#6768 StartFinishScope net472 871ns 0.183ns 0.684ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 654ns 0.411ns 1.48ns 0.0098 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 943ns 1.32ns 5.12ns 0.00899 0 0 696 B
master RunOnMethodBegin net472 1.12μs 0.417ns 1.62ns 0.104 0 0 658 B
#6768 RunOnMethodBegin net6.0 712ns 0.151ns 0.585ns 0.00961 0 0 696 B
#6768 RunOnMethodBegin netcoreapp3.1 860ns 0.327ns 1.27ns 0.00932 0 0 696 B
#6768 RunOnMethodBegin net472 1.2μs 0.382ns 1.48ns 0.104 0 0 658 B

@e-n-0 e-n-0 merged commit f40cb8c into master Mar 14, 2025
132 of 134 checks passed
@e-n-0 e-n-0 deleted the flavien/fix-system-tests-ci branch March 14, 2025 14:07
@github-actions github-actions bot added this to the vNext-v3 milestone Mar 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewlock andrewlock

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.13.0
Development

Successfully merging this pull request may close these issues.
2 participants
@e-n-0 @andrewlock