Simplify GitHub workflows, and add action to suggest changes instead of requiring manual updates #6764

andrewlock · 2025-03-13T12:12:52Z

Summary of changes

Unify multiple similar workflows
Add an action that will suggest the changes on the PR

Reason for change

We're currently building the tracer multiple times in different actions which is a bit excessive in terms of resource usage. We're also building the native components which isn't necessary (and slow).

Adding the suggested changes means that you don't have to run all this again locally just to commit the changes that it knows it needs

Implementation details

Cut, paste, and add a new action

Test coverage

This PR is the test

Other details

I think we could/should consider removing the "regenerate solution" from BuildTracerHome after this, as it removes a big part of the friction (and is annoyingly slow, at the start of a build)

… code (we don't need it) and add an action that posts suggestions to the PR

datadog-datadog-prod-us1 · 2025-03-13T12:15:04Z

Datadog Report

Branch report: andrew/ci/suggest-changes
Commit report: 2c88689
Test service: dd-trace-dotnet

✅ 0 Failed, 249726 Passed, 2396 Skipped, 19h 40m 29.42s Total Time

andrewlock · 2025-03-13T13:12:38Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (69ms)  : 67, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 71
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (1,001ms)  : 982, 1021
     .   : milestone, 1001,
    master - mean (1,002ms)  : 982, 1022
     .   : milestone, 1002,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (102ms)  : 100, 105
     .   : milestone, 102,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (681ms)  : 665, 697
     .   : milestone, 681,
    master - mean (685ms)  : 667, 703
     .   : milestone, 685,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (89ms)  : 87, 91
     .   : milestone, 89,
    master - mean (89ms)  : 86, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (639ms)  : 624, 654
     .   : milestone, 639,
    master - mean (643ms)  : 623, 663
     .   : milestone, 643,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (190ms)  : 186, 193
     .   : milestone, 190,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (1,101ms)  : 1078, 1124
     .   : milestone, 1101,
    master - mean (1,102ms)  : 1073, 1131
     .   : milestone, 1102,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (269ms)  : 266, 273
     .   : milestone, 269,
    master - mean (269ms)  : 263, 275
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (872ms)  : 833, 911
     .   : milestone, 872,
    master - mean (874ms)  : 836, 912
     .   : milestone, 874,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6764) - mean (262ms)  : 258, 265
     .   : milestone, 262,
    master - mean (262ms)  : 256, 267
     .   : milestone, 262,

    section CallTarget+Inlining+NGEN
    This PR (6764) - mean (851ms)  : 815, 888
     .   : milestone, 851,
    master - mean (858ms)  : 824, 892
     .   : milestone, 858,

andrewlock · 2025-03-13T13:34:15Z

Benchmarks Report for tracer 🐌

Benchmarks for #6764 compared to master:

1 benchmarks are faster, with geometric mean 1.157
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.08μs	45.8ns	318ns	0.0116	0.00387	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.1μs	54ns	275ns	0.0147	0.0049	0	5.8 KB
master	`StartStopWithChild`	net472	16μs	32.4ns	125ns	1.04	0.302	0.0953	6.21 KB
#6764	`StartStopWithChild`	net6.0	7.99μs	42.9ns	250ns	0.0224	0.0112	0	5.61 KB
#6764	`StartStopWithChild`	netcoreapp3.1	10.4μs	54.6ns	299ns	0.0157	0.00522	0	5.81 KB
#6764	`StartStopWithChild`	net472	16.1μs	62.9ns	243ns	1.06	0.333	0.0976	6.21 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	502μs	532ns	2.06μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	651μs	706ns	2.73μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	845μs	877ns	3.4μs	0.422	3.3 KB
#6764	`WriteAndFlushEnrichedTraces`	net6.0	513μs	276ns	1.03μs	0	2.7 KB
#6764	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	637μs	363ns	1.4μs	0	2.7 KB
#6764	`WriteAndFlushEnrichedTraces`	net472	862μs	830ns	3.21μs	0.428	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	135μs	287ns	1.07μs	0.134	14.47 KB
master	`SendRequest`	netcoreapp3.1	149μs	258ns	997ns	0.223	17.27 KB
master	`SendRequest`	net472	0.00108ns	0.000391ns	0.00146ns	0	0 b
#6764	`SendRequest`	net6.0	130μs	612ns	2.37μs	0.199	14.47 KB
#6764	`SendRequest`	netcoreapp3.1	148μs	319ns	1.24μs	0.224	17.27 KB
#6764	`SendRequest`	net472	0.000535ns	0.000228ns	0.000855ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #6764

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	41.7 KB	41.93 KB	235 B	0.56%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	567μs	2.67μs	10.3μs	0.566	0	0	41.66 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	693μs	3.73μs	21.7μs	0.361	0	0	41.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	845μs	3.86μs	14.9μs	8.81	2.52	0.419	53.31 KB
#6764	`WriteAndFlushEnrichedTraces`	net6.0	556μs	3.04μs	21.1μs	0.556	0	0	41.5 KB
#6764	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	698μs	4μs	29.1μs	0.357	0	0	41.93 KB
#6764	`WriteAndFlushEnrichedTraces`	net472	844μs	3.97μs	15.9μs	8.12	2.44	0.406	53.35 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.41μs	2.29ns	8.88ns	0.0139	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.8μs	1.48ns	5.52ns	0.0135	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.2μs	2.86ns	11.1ns	0.156	0.00109	987 B
#6764	`ExecuteNonQuery`	net6.0	1.34μs	2.13ns	8.24ns	0.0145	0	1.02 KB
#6764	`ExecuteNonQuery`	netcoreapp3.1	1.78μs	2.62ns	10.1ns	0.0132	0	1.02 KB
#6764	`ExecuteNonQuery`	net472	2.08μs	4.08ns	15.3ns	0.157	0.00103	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.25μs	1.05ns	3.77ns	0.0138	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.58μs	1.17ns	4.51ns	0.0125	976 B
master	`CallElasticsearch`	net472	2.62μs	2.15ns	8.32ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.32μs	0.874ns	3.39ns	0.0133	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.7μs	0.724ns	2.8ns	0.0136	1.02 KB
master	`CallElasticsearchAsync`	net472	2.64μs	3.54ns	13.7ns	0.166	1.05 KB
#6764	`CallElasticsearch`	net6.0	1.3μs	1.85ns	6.68ns	0.0136	976 B
#6764	`CallElasticsearch`	netcoreapp3.1	1.59μs	0.837ns	3.13ns	0.0133	976 B
#6764	`CallElasticsearch`	net472	2.57μs	0.847ns	3.28ns	0.158	995 B
#6764	`CallElasticsearchAsync`	net6.0	1.25μs	0.573ns	2.15ns	0.0131	952 B
#6764	`CallElasticsearchAsync`	netcoreapp3.1	1.66μs	0.524ns	1.89ns	0.0142	1.02 KB
#6764	`CallElasticsearchAsync`	net472	2.66μs	1.26ns	4.9ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.36μs	3.04ns	11.8ns	0.0136	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.66μs	0.994ns	3.72ns	0.0124	952 B
master	`ExecuteAsync`	net472	1.81μs	0.574ns	2.15ns	0.145	915 B
#6764	`ExecuteAsync`	net6.0	1.3μs	2.8ns	10.5ns	0.0136	952 B
#6764	`ExecuteAsync`	netcoreapp3.1	1.57μs	1.08ns	4.03ns	0.0125	952 B
#6764	`ExecuteAsync`	net472	1.81μs	0.585ns	2.26ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.35μs	1.86ns	6.95ns	0.0325	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.5μs	1.92ns	7.44ns	0.0384	2.85 KB
master	`SendAsync`	net472	7.48μs	1.81ns	7ns	0.494	3.12 KB
#6764	`SendAsync`	net6.0	4.49μs	1.71ns	6.15ns	0.0313	2.31 KB
#6764	`SendAsync`	netcoreapp3.1	5.32μs	2.36ns	8.51ns	0.0372	2.85 KB
#6764	`SendAsync`	net472	7.5μs	1.63ns	6.08ns	0.495	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #6764

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑netcoreapp3.1	1.157	2,423.58	2,095.22

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.5μs	0.728ns	2.73ns	0.0229	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.42μs	1.2ns	4.63ns	0.0218	1.64 KB
master	`EnrichedLog`	net472	2.42μs	0.772ns	2.99ns	0.249	1.57 KB
#6764	`EnrichedLog`	net6.0	1.46μs	0.555ns	2.08ns	0.0231	1.64 KB
#6764	`EnrichedLog`	netcoreapp3.1	2.1μs	0.686ns	2.47ns	0.0222	1.64 KB
#6764	`EnrichedLog`	net472	2.5μs	0.571ns	2.06ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	111μs	158ns	593ns	0.055	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	117μs	198ns	767ns	0	0	4.28 KB
master	`EnrichedLog`	net472	149μs	153ns	591ns	0.67	0.223	4.46 KB
#6764	`EnrichedLog`	net6.0	111μs	134ns	518ns	0	0	4.28 KB
#6764	`EnrichedLog`	netcoreapp3.1	117μs	93.6ns	363ns	0	0	4.28 KB
#6764	`EnrichedLog`	net472	149μs	161ns	582ns	0.67	0.223	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	3μs	0.605ns	2.26ns	0.03	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.29μs	3.24ns	12.5ns	0.03	2.2 KB
master	`EnrichedLog`	net472	4.83μs	1.47ns	5.71ns	0.321	2.02 KB
#6764	`EnrichedLog`	net6.0	3.05μs	1.28ns	4.97ns	0.0306	2.2 KB
#6764	`EnrichedLog`	netcoreapp3.1	4.28μs	2.16ns	8.36ns	0.0299	2.2 KB
#6764	`EnrichedLog`	net472	4.86μs	1.32ns	4.77ns	0.32	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.35μs	0.731ns	2.83ns	0.0156	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.79μs	1.35ns	5.23ns	0.0152	1.14 KB
master	`SendReceive`	net472	2.09μs	1ns	3.88ns	0.183	1.16 KB
#6764	`SendReceive`	net6.0	1.44μs	0.981ns	3.8ns	0.0158	1.14 KB
#6764	`SendReceive`	netcoreapp3.1	1.76μs	1.66ns	6.43ns	0.0149	1.14 KB
#6764	`SendReceive`	net472	2.06μs	0.458ns	1.77ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.77μs	1.32ns	4.95ns	0.0222	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.89μs	4.18ns	16.2ns	0.0213	1.65 KB
master	`EnrichedLog`	net472	4.43μs	1.91ns	7.4ns	0.323	2.04 KB
#6764	`EnrichedLog`	net6.0	2.77μs	0.855ns	3.31ns	0.0221	1.6 KB
#6764	`EnrichedLog`	netcoreapp3.1	3.93μs	2.79ns	10.4ns	0.0215	1.65 KB
#6764	`EnrichedLog`	net472	4.19μs	1.67ns	6.48ns	0.322	2.04 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	455ns	0.131ns	0.506ns	0.00803	576 B
master	`StartFinishSpan`	netcoreapp3.1	595ns	0.381ns	1.48ns	0.00772	576 B
master	`StartFinishSpan`	net472	688ns	1.76ns	6.8ns	0.0918	578 B
master	`StartFinishScope`	net6.0	484ns	0.0714ns	0.277ns	0.00988	696 B
master	`StartFinishScope`	netcoreapp3.1	776ns	0.455ns	1.76ns	0.00936	696 B
master	`StartFinishScope`	net472	850ns	0.291ns	1.13ns	0.104	658 B
#6764	`StartFinishSpan`	net6.0	410ns	0.173ns	0.67ns	0.00801	576 B
#6764	`StartFinishSpan`	netcoreapp3.1	604ns	0.138ns	0.497ns	0.00785	576 B
#6764	`StartFinishSpan`	net472	642ns	0.384ns	1.49ns	0.0917	578 B
#6764	`StartFinishScope`	net6.0	534ns	0.271ns	1.05ns	0.00969	696 B
#6764	`StartFinishScope`	netcoreapp3.1	749ns	0.227ns	0.818ns	0.00938	696 B
#6764	`StartFinishScope`	net472	874ns	0.196ns	0.76ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	654ns	0.411ns	1.48ns	0.0098	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	943ns	1.32ns	5.12ns	0.00899	696 B
master	`RunOnMethodBegin`	net472	1.12μs	0.417ns	1.62ns	0.104	658 B
#6764	`RunOnMethodBegin`	net6.0	714ns	0.544ns	1.96ns	0.00997	696 B
#6764	`RunOnMethodBegin`	netcoreapp3.1	989ns	0.919ns	3.56ns	0.00899	696 B
#6764	`RunOnMethodBegin`	net472	1.17μs	0.414ns	1.6ns	0.104	658 B

andrewlock added 4 commits March 12, 2025 17:34

Combine multiple workflows into one, don't bother building the native…

c5eee4b

… code (we don't need it) and add an action that posts suggestions to the PR

Fix generated solution diff

901deed

Default to comment instead, to avoid needing to discard review

2333059

TEMP: add new reference that needs trimming

e8c1afa

andrewlock added the area:builds label Mar 13, 2025

Not sure why this isn't working now...

0bebbee

Try adding debug logs

2c88689

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify GitHub workflows, and add action to suggest changes instead of requiring manual updates #6764

Simplify GitHub workflows, and add action to suggest changes instead of requiring manual updates #6764

andrewlock commented Mar 13, 2025

datadog-datadog-prod-us1 bot commented Mar 13, 2025 •

edited

Loading

andrewlock commented Mar 13, 2025 •

edited

Loading

andrewlock commented Mar 13, 2025 •

edited

Loading

Raw results

Raw results

Raw results

More allocations ⚠️ in #6764

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6764

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Simplify GitHub workflows, and add action to suggest changes instead of requiring manual updates #6764

Are you sure you want to change the base?

Simplify GitHub workflows, and add action to suggest changes instead of requiring manual updates #6764

Conversation

andrewlock commented Mar 13, 2025

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-datadog-prod-us1 bot commented Mar 13, 2025 • edited Loading

Datadog Report

andrewlock commented Mar 13, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Mar 13, 2025 • edited Loading

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

More allocations ⚠️ in #6764

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #6764

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

datadog-datadog-prod-us1 bot commented Mar 13, 2025 •

edited

Loading

andrewlock commented Mar 13, 2025 •

edited

Loading

andrewlock commented Mar 13, 2025 •

edited

Loading