Add a retry to the macos unit tests #4571

andrewlock · 2023-08-31T13:06:38Z

Summary of changes

Adds a retry on failure to the unit_tests_macos stage

Reason for change

macOS builds are just flaky. My latest bugbear

Unhandled exception: System.ComponentModel.Win32Exception (13): An error occurred trying to start process '/Users/runner/work/1/s/tracer/build/_build/bin/Debug/_build' with working directory '/Users/runner/work/1/s'. Permission denied
   at System.Diagnostics.Process.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, String[] envp, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at Microsoft.DotNet.Cli.Utils.Command.Execute(Action`1 processStarted)
   at Microsoft.DotNet.Tools.Run.RunCommand.Execute()
   at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DotNet.Cli.Parser.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<<UseHelp>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__17_0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass15_0.<<UseParseDirective>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass7_0.<<UseExceptionHandler>b__0>d.MoveNext()

~~Tony thinks this happened since the latest Nuke update, but I'm not sure why (or if that's definitely the case).~~ It's a .NET CLI issue 🤷‍♂️

Implementation details

Crude retry of the stage. This will annoyingly also retry the stage if the unit tests fail, but that doesn't happen often enough to become a big issue IMO, and this is more annoying.

Test coverage

Time will tell

andrewlock · 2023-08-31T13:29:06Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (72ms)  : 60, 83
     .   : milestone, 72,
    master - mean (77ms)  : 65, 89
     .   : milestone, 77,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (1,039ms)  : 999, 1079
     .   : milestone, 1039,
    master - mean (1,035ms)  : 996, 1075
     .   : milestone, 1035,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (106ms)  : 101, 111
     .   : milestone, 106,
    master - mean (106ms)  : 99, 114
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (732ms)  : 702, 762
     .   : milestone, 732,
    master - mean (729ms)  : 692, 766
     .   : milestone, 729,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (90ms)  : 84, 96
     .   : milestone, 90,
    master - mean (89ms)  : 87, 91
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (690ms)  : 660, 720
     .   : milestone, 690,
    master - mean (688ms)  : 655, 722
     .   : milestone, 688,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (192ms)  : 187, 197
     .   : milestone, 192,
    master - mean (189ms)  : 182, 195
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (1,135ms)  : 1097, 1172
     .   : milestone, 1135,
    master - mean (1,129ms)  : 1098, 1159
     .   : milestone, 1129,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (276ms)  : 268, 284
     .   : milestone, 276,
    master - mean (269ms)  : 259, 279
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (1,093ms)  : 1064, 1122
     .   : milestone, 1093,
    master - mean (1,072ms)  : 1045, 1099
     .   : milestone, 1072,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4571) - mean (265ms)  : 259, 271
     .   : milestone, 265,
    master - mean (259ms)  : 250, 267
     .   : milestone, 259,

    section CallTarget+Inlining+NGEN
    This PR (4571) - mean (1,055ms)  : 1030, 1081
     .   : milestone, 1055,
    master - mean (1,047ms)  : 1018, 1075
     .   : milestone, 1047,

datadog-ddstaging · 2023-08-31T13:35:44Z

Datadog Report

Branch report: andrew/ci/shakes-fist-at-macos
Commit report: 93e3c66

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 299659 Passed, 1115 Skipped, 35m 38.42s Wall Time

tonyredondo · 2023-08-31T13:36:33Z

I don't think is nuke but the dotnet CLI itself, we are not the only one suffering from this: dotnet/sdk#10237,
dotnet/runtime#52343

andrewlock · 2023-08-31T13:57:48Z

Benchmarks Report 🐌

Benchmarks for #4571 compared to master:

2 benchmarks are faster, with geometric mean 1.228
3 benchmarks are slower, with geometric mean 1.149
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.22μs	45.3ns	276ns	0.0241	0.0121	0	7.3 KB
master	`StartStopWithChild`	netcoreapp3.1	9.92μs	52.2ns	276ns	0.0288	0.0144	0	7.39 KB
master	`StartStopWithChild`	net472	15.9μs	51ns	197ns	1.28	0.31	0.101	7.67 KB
#4571	`StartStopWithChild`	net6.0	8.22μs	42.3ns	207ns	0.0202	0.00808	0	7.3 KB
#4571	`StartStopWithChild`	netcoreapp3.1	9.9μs	54.1ns	301ns	0.025	0.005	0	7.4 KB
#4571	`StartStopWithChild`	net472	16.1μs	43ns	167ns	1.28	0.329	0.102	7.67 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	489μs	147ns	571ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	635μs	587ns	2.19μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	800μs	364ns	1.41μs	0.398	3.3 KB
#4571	`WriteAndFlushEnrichedTraces`	net6.0	472μs	355ns	1.33μs	0	2.7 KB
#4571	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	638μs	446ns	1.67μs	0	2.7 KB
#4571	`WriteAndFlushEnrichedTraces`	net472	790μs	240ns	930ns	0.396	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4571

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1	1.337	228.34	170.78

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	39.3μs	55ns	206ns	0.0197	0	1.69 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	43.2μs	143ns	553ns	0.0215	0	1.67 KB
master	`AllCycleSimpleBody`	net472	43μs	52.3ns	202ns	0.275	0	1.73 KB
master	`AllCycleMoreComplexBody`	net6.0	234μs	668ns	3.34μs	0.116	0	9.26 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	242μs	144ns	558ns	0.12	0	9.16 KB
master	`AllCycleMoreComplexBody`	net472	252μs	50ns	194ns	1.37	0	9.33 KB
master	`ObjectExtractorSimpleBody`	net6.0	120ns	0.033ns	0.123ns	0.00396	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	228ns	0.0549ns	0.206ns	0.00373	0	272 B
master	`ObjectExtractorSimpleBody`	net472	146ns	0.105ns	0.392ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	3μs	1.07ns	4.15ns	0.054	0	3.88 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.09μs	1.37ns	4.94ns	0.0508	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.21μs	4.88ns	18.9ns	0.617	0.00634	3.89 KB
#4571	`AllCycleSimpleBody`	net6.0	39.2μs	79.8ns	299ns	0.02	0	1.69 KB
#4571	`AllCycleSimpleBody`	netcoreapp3.1	40.7μs	27ns	97.2ns	0.0203	0	1.67 KB
#4571	`AllCycleSimpleBody`	net472	42.6μs	91ns	353ns	0.274	0	1.73 KB
#4571	`AllCycleMoreComplexBody`	net6.0	234μs	191ns	739ns	0.116	0	9.26 KB
#4571	`AllCycleMoreComplexBody`	netcoreapp3.1	240μs	275ns	1.07μs	0.12	0	9.16 KB
#4571	`AllCycleMoreComplexBody`	net472	249μs	145ns	542ns	1.48	0	9.33 KB
#4571	`ObjectExtractorSimpleBody`	net6.0	119ns	0.0561ns	0.21ns	0.00394	0	280 B
#4571	`ObjectExtractorSimpleBody`	netcoreapp3.1	171ns	0.0572ns	0.214ns	0.0037	0	272 B
#4571	`ObjectExtractorSimpleBody`	net472	146ns	0.136ns	0.527ns	0.0446	0	281 B
#4571	`ObjectExtractorMoreComplexBody`	net6.0	3μs	0.629ns	2.35ns	0.0539	0	3.88 KB
#4571	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.04μs	1.19ns	4.45ns	0.0526	0	3.78 KB
#4571	`ObjectExtractorMoreComplexBody`	net472	4.08μs	3.16ns	12.2ns	0.617	0.00611	3.89 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	122μs	980ns	9.35μs	17.62 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	129μs	608ns	2.36μs	16.79 KB
master	`RunWaf(args=NestedMap (10))`	net472	131μs	709ns	3.68μs	24.58 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	238μs	1.65μs	15.9μs	37.95 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	260μs	1.22μs	4.57μs	37.78 KB
master	`RunWaf(args=NestedMap (100))`	net472	253μs	1.27μs	5.97μs	40.96 KB
master	`RunWaf(args=NestedMap (1000))`	net6.0	238μs	1.22μs	5.7μs	37.95 KB
master	`RunWaf(args=NestedMap (1000))`	netcoreapp3.1	258μs	1.43μs	8.19μs	37.78 KB
master	`RunWaf(args=NestedMap (1000))`	net472	260μs	1.15μs	4.31μs	40.96 KB
#4571	`RunWaf(args=NestedMap (10))`	net6.0	120μs	639ns	4.99μs	17.62 KB
#4571	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	131μs	754ns	5.79μs	16.79 KB
#4571	`RunWaf(args=NestedMap (10))`	net472	132μs	737ns	4.48μs	24.58 KB
#4571	`RunWaf(args=NestedMap (100))`	net6.0	230μs	700ns	3.28μs	37.95 KB
#4571	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	261μs	1.44μs	8.14μs	37.78 KB
#4571	`RunWaf(args=NestedMap (100))`	net472	268μs	1.41μs	9.11μs	40.96 KB
#4571	`RunWaf(args=NestedMap (1000))`	net6.0	248μs	2.31μs	22.3μs	37.95 KB
#4571	`RunWaf(args=NestedMap (1000))`	netcoreapp3.1	284μs	1.67μs	15.4μs	37.78 KB
#4571	`RunWaf(args=NestedMap (1000))`	net472	266μs	1.5μs	10.6μs	40.96 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	169μs	184ns	713ns	0.169	18.09 KB
master	`SendRequest`	netcoreapp3.1	188μs	356ns	1.38μs	0.188	20.25 KB
master	`SendRequest`	net472	0.00176ns	0.000513ns	0.00199ns	0	0 b
#4571	`SendRequest`	net6.0	170μs	143ns	554ns	0.17	18.09 KB
#4571	`SendRequest`	netcoreapp3.1	191μs	213ns	797ns	0.191	20.25 KB
#4571	`SendRequest`	net472	0.000918ns	0.000315ns	0.00122ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	507μs	274ns	1.03μs	0.498	0	0	41.85 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	615μs	1μs	3.87μs	0.309	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	net472	764μs	2.25μs	8.7μs	8.38	2.67	0.381	53.25 KB
#4571	`WriteAndFlushEnrichedTraces`	net6.0	514μs	664ns	2.3μs	0.534	0	0	41.82 KB
#4571	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	612μs	1.53μs	5.94μs	0.303	0	0	41.6 KB
#4571	`WriteAndFlushEnrichedTraces`	net472	782μs	2.23μs	8.65μs	8.15	2.33	0.388	53.23 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.16μs	0.662ns	2.56ns	0.0104	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.38μs	2.15ns	8.32ns	0.0103	768 B
master	`ExecuteNonQuery`	net472	1.66μs	0.466ns	1.61ns	0.115	730 B
#4571	`ExecuteNonQuery`	net6.0	1.16μs	0.769ns	2.98ns	0.0108	768 B
#4571	`ExecuteNonQuery`	netcoreapp3.1	1.25μs	0.593ns	2.14ns	0.0106	768 B
#4571	`ExecuteNonQuery`	net472	1.64μs	3.9ns	15.1ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.14μs	0.694ns	2.6ns	0.0138	0	992 B
master	`CallElasticsearch`	netcoreapp3.1	1.49μs	0.752ns	2.81ns	0.0134	0	992 B
master	`CallElasticsearch`	net472	2.41μs	0.638ns	2.47ns	0.159	0.00121	1 KB
master	`CallElasticsearchAsync`	net6.0	1.37μs	0.74ns	2.87ns	0.0137	0	968 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.55μs	0.835ns	3.13ns	0.014	0	1.04 KB
master	`CallElasticsearchAsync`	net472	2.62μs	0.921ns	3.44ns	0.167	0.00132	1.06 KB
#4571	`CallElasticsearch`	net6.0	1.17μs	0.451ns	1.69ns	0.0135	0	992 B
#4571	`CallElasticsearch`	netcoreapp3.1	1.53μs	0.894ns	3.35ns	0.013	0	992 B
#4571	`CallElasticsearch`	net472	2.34μs	0.75ns	2.81ns	0.159	0.00117	1 KB
#4571	`CallElasticsearchAsync`	net6.0	1.31μs	0.341ns	1.28ns	0.0136	0	968 B
#4571	`CallElasticsearchAsync`	netcoreapp3.1	1.55μs	0.554ns	2.15ns	0.014	0	1.04 KB
#4571	`CallElasticsearchAsync`	net472	2.49μs	0.789ns	3.06ns	0.168	0.00126	1.06 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net6.0	1.21μs	0.385ns	1.44ns	0.0127	0	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.53μs	4.28ns	16.6ns	0.0121	0	912 B
master	`ExecuteAsync`	net472	1.61μs	1.1ns	4.27ns	0.139	0.000798	875 B
#4571	`ExecuteAsync`	net6.0	1.24μs	0.497ns	1.92ns	0.0129	0	912 B
#4571	`ExecuteAsync`	netcoreapp3.1	1.52μs	2.2ns	8.22ns	0.012	0	912 B
#4571	`ExecuteAsync`	net472	1.65μs	0.937ns	3.63ns	0.139	0.000826	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	3.7μs	1.35ns	5.07ns	0.0259	1.94 KB
master	`SendAsync`	netcoreapp3.1	4.52μs	1.55ns	5.82ns	0.034	2.48 KB
master	`SendAsync`	net472	7.17μs	4.79ns	17.9ns	0.484	3.05 KB
#4571	`SendAsync`	net6.0	3.86μs	0.998ns	3.87ns	0.0271	1.94 KB
#4571	`SendAsync`	netcoreapp3.1	4.7μs	2.12ns	7.92ns	0.0327	2.48 KB
#4571	`SendAsync`	net472	7.07μs	3.26ns	12.2ns	0.483	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.34μs	0.465ns	1.74ns	0.0228	1.62 KB
master	`EnrichedLog`	netcoreapp3.1	2.06μs	1.41ns	5.28ns	0.0218	1.62 KB
master	`EnrichedLog`	net472	2.41μs	3.62ns	14ns	0.244	1.54 KB
#4571	`EnrichedLog`	net6.0	1.38μs	0.656ns	2.45ns	0.0229	1.62 KB
#4571	`EnrichedLog`	netcoreapp3.1	2.03μs	0.71ns	2.66ns	0.0215	1.62 KB
#4571	`EnrichedLog`	net472	2.27μs	1.11ns	4.3ns	0.244	1.54 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	112μs	213ns	825ns	0.0553	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	167ns	623ns	0	0	4.21 KB
master	`EnrichedLog`	net472	146μs	68.1ns	255ns	0.654	0.218	4.38 KB
#4571	`EnrichedLog`	net6.0	111μs	158ns	613ns	0.0557	0	4.21 KB
#4571	`EnrichedLog`	netcoreapp3.1	118μs	120ns	465ns	0	0	4.21 KB
#4571	`EnrichedLog`	net472	145μs	62ns	232ns	0.654	0.218	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.93μs	0.93ns	3.48ns	0.0309	2.18 KB
master	`EnrichedLog`	netcoreapp3.1	3.93μs	1.12ns	4.33ns	0.0295	2.18 KB
master	`EnrichedLog`	net472	4.65μs	2.04ns	7.9ns	0.315	1.99 KB
#4571	`EnrichedLog`	net6.0	2.96μs	0.958ns	3.45ns	0.031	2.18 KB
#4571	`EnrichedLog`	netcoreapp3.1	3.94μs	2.87ns	10.8ns	0.0276	2.18 KB
#4571	`EnrichedLog`	net472	4.66μs	2.43ns	9.41ns	0.314	1.99 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`SendReceive`	net6.0	1.25μs	0.602ns	2.33ns	0.0163	0	1.16 KB
master	`SendReceive`	netcoreapp3.1	1.65μs	0.939ns	3.51ns	0.0156	0	1.16 KB
master	`SendReceive`	net472	2.02μs	2.58ns	9.66ns	0.185	0	1.16 KB
#4571	`SendReceive`	net6.0	1.26μs	0.773ns	2.89ns	0.0163	0	1.16 KB
#4571	`SendReceive`	netcoreapp3.1	1.72μs	0.634ns	2.46ns	0.0154	0	1.16 KB
#4571	`SendReceive`	net472	1.95μs	0.564ns	2.03ns	0.185	0.000973	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.58μs	1.1ns	4.11ns	0.022	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.46μs	1.54ns	5.56ns	0.0207	1.58 KB
master	`EnrichedLog`	net472	4.18μs	1.36ns	5.08ns	0.31	1.96 KB
#4571	`EnrichedLog`	net6.0	2.56μs	0.823ns	2.97ns	0.0219	1.53 KB
#4571	`EnrichedLog`	netcoreapp3.1	3.53μs	1.39ns	5.19ns	0.0212	1.58 KB
#4571	`EnrichedLog`	net472	4.04μs	0.841ns	3.03ns	0.311	1.96 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4571

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1	1.155	516.11	595.91
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472	1.133	636.68	721.59

Faster 🎉 in #4571

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.128	534.15	473.43

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	461ns	0.173ns	0.647ns	0.00741	536 B
master	`StartFinishSpan`	netcoreapp3.1	516ns	0.249ns	0.932ns	0.00709	536 B
master	`StartFinishSpan`	net472	637ns	0.173ns	0.669ns	0.0852	538 B
master	`StartFinishScope`	net6.0	534ns	0.188ns	0.728ns	0.00929	656 B
master	`StartFinishScope`	netcoreapp3.1	750ns	0.481ns	1.86ns	0.00874	656 B
master	`StartFinishScope`	net472	863ns	0.679ns	2.63ns	0.0982	618 B
#4571	`StartFinishSpan`	net6.0	457ns	0.164ns	0.612ns	0.00759	536 B
#4571	`StartFinishSpan`	netcoreapp3.1	596ns	0.197ns	0.711ns	0.00717	536 B
#4571	`StartFinishSpan`	net472	722ns	0.154ns	0.576ns	0.0852	538 B
#4571	`StartFinishScope`	net6.0	474ns	0.216ns	0.836ns	0.00915	656 B
#4571	`StartFinishScope`	netcoreapp3.1	750ns	0.235ns	0.879ns	0.00901	656 B
#4571	`StartFinishScope`	net472	842ns	0.193ns	0.695ns	0.098	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #4571

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0	1.159	511.17	592.62

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	511ns	0.126ns	0.49ns	0.00922	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	845ns	0.406ns	1.46ns	0.00898	656 B
master	`RunOnMethodBegin`	net472	982ns	0.159ns	0.594ns	0.0978	618 B
#4571	`RunOnMethodBegin`	net6.0	593ns	0.181ns	0.677ns	0.00918	656 B
#4571	`RunOnMethodBegin`	netcoreapp3.1	815ns	0.219ns	0.82ns	0.00867	656 B
#4571	`RunOnMethodBegin`	net472	995ns	0.19ns	0.684ns	0.0981	618 B

andrewlock · 2023-08-31T16:17:38Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4571) (10.846M)   : 0, 10846134
    master (10.889M)   : 0, 10889070
    benchmarks/2.36.0 (11.139M)   : 0, 11139004
    benchmarks/2.9.0 (11.325M)   : 0, 11324702

    section Automatic
    This PR (4571) (7.677M)   : 0, 7677480
    master (7.589M)   : 0, 7589078
    benchmarks/2.36.0 (7.796M)   : 0, 7795573
    benchmarks/2.9.0 (8.180M)   : 0, 8179828

    section Trace stats
    master (7.518M)   : 0, 7517507
    benchmarks/2.36.0 (7.932M)   : 0, 7931920

    section Manual
    This PR (4571) (9.636M)   : 0, 9635578
    master (9.729M)   : 0, 9728533
    benchmarks/2.36.0 (9.854M)   : 0, 9854091

    section Manual + Automatic
    This PR (4571) (7.283M)   : 0, 7282771
    master (7.310M)   : 0, 7310496
    benchmarks/2.36.0 (7.368M)   : 0, 7368211

    section Version Conflict
    master (6.537M)   : 0, 6536737
    benchmarks/2.36.0 (6.826M)   : 0, 6825561

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4571) (9.729M)   : 0, 9728847
    master (9.586M)   : 0, 9585597
    benchmarks/2.36.0 (9.434M)   : 0, 9433740
    benchmarks/2.9.0 (9.767M)   : 0, 9766880

    section Automatic
    This PR (4571) (6.660M)   : 0, 6660477
    master (6.767M)   : 0, 6766915
    benchmarks/2.36.0 (6.695M)   : 0, 6694737

    section Trace stats
    master (6.766M)   : 0, 6765987
    benchmarks/2.36.0 (6.914M)   : 0, 6914146

    section Manual
    This PR (4571) (8.407M)   : 0, 8406557
    master (8.384M)   : 0, 8384071
    benchmarks/2.36.0 (8.612M)   : 0, 8612005

    section Manual + Automatic
    This PR (4571) (6.500M)   : 0, 6499790
    master (6.546M)   : 0, 6546407
    benchmarks/2.36.0 (6.562M)   : 0, 6562387

    section Version Conflict
    master (5.915M)   : 0, 5914883
    benchmarks/2.36.0 (5.848M)   : 0, 5847902

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4571) (10.598M)   : 0, 10597707
    master (10.368M)   : 0, 10368399
    benchmarks/2.36.0 (10.442M)   : 0, 10442160
    benchmarks/2.9.0 (10.288M)   : 0, 10288053

    section Automatic
    This PR (4571) (7.393M)   : 0, 7393233
    master (7.298M)   : 0, 7298398
    benchmarks/2.36.0 (7.461M)   : 0, 7460996
    benchmarks/2.9.0 (7.532M)   : 0, 7532387

    section Trace stats
    master (7.155M)   : 0, 7154905
    benchmarks/2.36.0 (7.459M)   : 0, 7458598

    section Manual
    This PR (4571) (9.138M)   : 0, 9137825
    master (8.790M)   : 0, 8789556
    benchmarks/2.36.0 (9.183M)   : 0, 9183081

    section Manual + Automatic
    This PR (4571) (7.194M)   : 0, 7193640
    master (7.088M)   : 0, 7088347
    benchmarks/2.36.0 (7.212M)   : 0, 7211940

    section Version Conflict
    master (6.609M)   : 0, 6608508
    benchmarks/2.36.0 (6.514M)   : 0, 6514071

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4571) (7.385M)   : 0, 7385328
    master (7.429M)   : 0, 7428840
    benchmarks/2.36.0 (7.497M)   : 0, 7496686
    benchmarks/2.9.0 (7.829M)   : 0, 7829234

    section No attack
    This PR (4571) (2.113M)   : 0, 2112810
    master (2.131M)   : 0, 2130760
    benchmarks/2.36.0 (2.139M)   : 0, 2138929
    benchmarks/2.9.0 (3.223M)   : 0, 3222946

    section Attack
    This PR (4571) (1.676M)   : 0, 1676000
    master (1.662M)   : 0, 1662210
    benchmarks/2.36.0 (1.679M)   : 0, 1678844
    benchmarks/2.9.0 (2.579M)   : 0, 2579343

    section Blocking
    This PR (4571) (3.094M)   : 0, 3094393
    master (3.131M)   : 0, 3131011
    benchmarks/2.36.0 (3.142M)   : 0, 3141981

Add a retry to the macos unit tests

93e3c66

andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Aug 31, 2023

andrewlock requested a review from a team as a code owner August 31, 2023 13:06

tonyredondo approved these changes Sep 1, 2023

View reviewed changes

zacharycmontoya approved these changes Sep 1, 2023

View reviewed changes

andrewlock merged commit 8dfe106 into master Sep 1, 2023
51 checks passed

andrewlock deleted the andrew/ci/shakes-fist-at-macos branch September 1, 2023 16:36

github-actions bot added this to the vNext milestone Sep 1, 2023

kevingosse mentioned this pull request Jan 30, 2024

Avoid using dotnet run in nuke #5127

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a retry to the macos unit tests #4571

Add a retry to the macos unit tests #4571

andrewlock commented Aug 31, 2023 •

edited

Loading

andrewlock commented Aug 31, 2023

datadog-ddstaging bot commented Aug 31, 2023 •

edited

Loading

tonyredondo commented Aug 31, 2023 •

edited

Loading

andrewlock commented Aug 31, 2023

Raw results

Raw results

Faster 🎉 in #4571

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4571

Faster 🎉 in #4571

Raw results

Slower ⚠️ in #4571

Raw results

andrewlock commented Aug 31, 2023

Add a retry to the macos unit tests #4571

Add a retry to the macos unit tests #4571

Conversation

andrewlock commented Aug 31, 2023 • edited Loading

Summary of changes

Reason for change

Implementation details

Test coverage

andrewlock commented Aug 31, 2023

Execution-Time Benchmarks Report ⏱️

datadog-ddstaging bot commented Aug 31, 2023 • edited Loading

Datadog Report

tonyredondo commented Aug 31, 2023 • edited Loading

andrewlock commented Aug 31, 2023

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Faster 🎉 in #4571

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #4571

Faster 🎉 in #4571

Raw results

Slower ⚠️ in #4571

Raw results

andrewlock commented Aug 31, 2023

Throughput/Crank Report:zap:

andrewlock commented Aug 31, 2023 •

edited

Loading

datadog-ddstaging bot commented Aug 31, 2023 •

edited

Loading

tonyredondo commented Aug 31, 2023 •

edited

Loading