Restrict sending APM-related metrics when running in CI app #4618

andrewlock · 2023-09-13T09:40:11Z

Summary of changes

Create a new metric collector implementation for CIApp
Only send ciapp-specific metrics (and some extra metrics of interest) when running in ciapp

Reason for change

When we enable v2 telemetry by default, ciapp executions will also send metrics. While this may be of interest, it could also just muddy the waters, as we don't currently have a way of differentiating between ciapp metrics and "normal" APM metrics.

I recently discussed with @yshapiro-57 about adding "app-type" headers to the telemetry requests. These could be used for multiple things, but one possibility might be for adding as a tag to all metrics received in a generate-metrics requests, which would save us the overhead and complexity of needing to add that tag to all the metrics ourselves. Either way, that's an enhancement we can add at a later date if we wish.

Implementation details

Extract the bulk of the MetricsTelemetryCollector implementation into a base class
Add a new CiVisibilityTelemetryCollector implementation that derives from the base class, and uses null Record*() methods for metrics that are not of interest
Update the CiVsibilityTracerManager to replace the global metrics collector.
- Note that currently, this means any metrics recorded to the collector prior to doing the replacement will be lost. I don't think this is an issue currently (we do the replacement early enough), and preserving them could be complicated, but we can revisit this later if needs be.

Test coverage

Added unit tests for the source generator. We don't currently have any ciapp metrics yet, so no integration tests. Depending which is merged first #4596, can be used to confirm this works as expected.

Other details

datadog-ddstaging · 2023-09-13T10:00:51Z

Datadog Report

Branch report: andrew/telemetry/ci-app-metrics
Commit report: c1bf698

✅ dd-trace-dotnet: 0 Failed, 0 New Flaky, 302183 Passed, 1154 Skipped, 22m 10.99s Wall Time

andrewlock · 2023-09-13T10:02:23Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (68ms)  : 65, 72
     .   : milestone, 68,
    master - mean (68ms)  : 65, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (1,040ms)  : 1017, 1062
     .   : milestone, 1040,
    master - mean (1,032ms)  : 1008, 1055
     .   : milestone, 1032,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (106ms)  : 101, 111
     .   : milestone, 106,
    master - mean (105ms)  : 97, 114
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (748ms)  : 728, 769
     .   : milestone, 748,
    master - mean (747ms)  : 727, 766
     .   : milestone, 747,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (90ms)  : 87, 92
     .   : milestone, 90,
    master - mean (89ms)  : 83, 95
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (709ms)  : 693, 726
     .   : milestone, 709,
    master - mean (718ms)  : 695, 741
     .   : milestone, 718,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (190ms)  : 182, 198
     .   : milestone, 190,
    master - mean (186ms)  : 178, 194
     .   : milestone, 186,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (1,137ms)  : 1112, 1163
     .   : milestone, 1137,
    master - mean (1,114ms)  : 1079, 1150
     .   : milestone, 1114,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (269ms)  : 257, 282
     .   : milestone, 269,
    master - mean (266ms)  : 254, 277
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (1,115ms)  : 1090, 1141
     .   : milestone, 1115,
    master - mean (1,095ms)  : 1056, 1134
     .   : milestone, 1095,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4618) - mean (263ms)  : 255, 271
     .   : milestone, 263,
    master - mean (258ms)  : 248, 268
     .   : milestone, 258,

    section CallTarget+Inlining+NGEN
    This PR (4618) - mean (1,064ms)  : 1037, 1091
     .   : milestone, 1064,
    master - mean (1,043ms)  : 1008, 1079
     .   : milestone, 1043,

andrewlock · 2023-09-13T10:22:49Z

Benchmarks Report 🐌

Benchmarks for #4618 compared to master:

1 benchmarks are faster, with geometric mean 1.156
1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.21μs	41.7ns	225ns	0.0237	0.0119	0	7.29 KB
master	`StartStopWithChild`	netcoreapp3.1	10.3μs	58ns	376ns	0.0258	0.0103	0	7.39 KB
master	`StartStopWithChild`	net472	15.6μs	55.7ns	216ns	1.28	0.308	0.0923	7.66 KB
#4618	`StartStopWithChild`	net6.0	8.03μs	40.3ns	171ns	0.0273	0.0117	0	7.29 KB
#4618	`StartStopWithChild`	netcoreapp3.1	9.97μs	55ns	356ns	0.0294	0.0147	0	7.38 KB
#4618	`StartStopWithChild`	net472	15.5μs	38.8ns	145ns	1.3	0.338	0.1	7.66 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	470μs	279ns	1.08μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	660μs	254ns	951ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	803μs	373ns	1.34μs	0.401	3.3 KB
#4618	`WriteAndFlushEnrichedTraces`	net6.0	477μs	245ns	949ns	0	2.7 KB
#4618	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	633μs	111ns	417ns	0	2.7 KB
#4618	`WriteAndFlushEnrichedTraces`	net472	795μs	376ns	1.41μs	0.398	3.3 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	42.8μs	79.7ns	287ns	0.0215	0	1.78 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	45.6μs	84.1ns	315ns	0	0	1.75 KB
master	`AllCycleSimpleBody`	net472	47.9μs	15.7ns	60.7ns	0.283	0	1.82 KB
master	`AllCycleMoreComplexBody`	net6.0	235μs	88.1ns	341ns	0.118	0	9.35 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	251μs	368ns	1.43μs	0	0	9.24 KB
master	`AllCycleMoreComplexBody`	net472	261μs	282ns	1.09μs	1.44	0	9.42 KB
master	`ObjectExtractorSimpleBody`	net6.0	116ns	0.0364ns	0.131ns	0.00395	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	175ns	0.178ns	0.691ns	0.00367	0	272 B
master	`ObjectExtractorSimpleBody`	net472	144ns	0.128ns	0.496ns	0.0446	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	2.94μs	0.56ns	2.1ns	0.0543	0	3.88 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.06μs	1.71ns	6.4ns	0.0508	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	net472	4.15μs	4.27ns	16ns	0.617	0.00615	3.89 KB
#4618	`AllCycleSimpleBody`	net6.0	43.4μs	29.2ns	113ns	0.0217	0	1.78 KB
#4618	`AllCycleSimpleBody`	netcoreapp3.1	46.4μs	136ns	526ns	0.0231	0	1.75 KB
#4618	`AllCycleSimpleBody`	net472	48.6μs	34.7ns	134ns	0.266	0	1.82 KB
#4618	`AllCycleMoreComplexBody`	net6.0	238μs	73.5ns	345ns	0.12	0	9.35 KB
#4618	`AllCycleMoreComplexBody`	netcoreapp3.1	247μs	129ns	465ns	0	0	9.24 KB
#4618	`AllCycleMoreComplexBody`	net472	264μs	133ns	626ns	1.45	0	9.42 KB
#4618	`ObjectExtractorSimpleBody`	net6.0	116ns	0.0429ns	0.161ns	0.00395	0	280 B
#4618	`ObjectExtractorSimpleBody`	netcoreapp3.1	167ns	0.101ns	0.379ns	0.00365	0	272 B
#4618	`ObjectExtractorSimpleBody`	net472	146ns	0.108ns	0.373ns	0.0446	0	281 B
#4618	`ObjectExtractorMoreComplexBody`	net6.0	3.08μs	0.885ns	3.31ns	0.0543	0	3.88 KB
#4618	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	4.04μs	5.88ns	22.8ns	0.0509	0	3.78 KB
#4618	`ObjectExtractorMoreComplexBody`	net472	4.18μs	2.54ns	9.51ns	0.618	0.00626	3.89 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`RunWaf(args=NestedMap (10))`	net6.0	49.5μs	182ns	629ns	0.219	0	16.07 KB
master	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	67.7μs	381ns	2.64μs	0.2	0	16.06 KB
master	`RunWaf(args=NestedMap (10))`	net472	96.1μs	35ns	121ns	2.54	0.0957	16.14 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	116μs	38.8ns	145ns	0.291	0	22.42 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	138μs	527ns	1.9μs	0.276	0	22.37 KB
master	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	166μs	75ns	291ns	3.54	0.165	22.71 KB
master	`RunWaf(args=NestedMap (100))`	net6.0	126μs	604ns	2.34μs	0.447	0	34.74 KB
master	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	165μs	864ns	4.14μs	0.469	0	35.39 KB
master	`RunWaf(args=NestedMap (100))`	net472	222μs	154ns	596ns	5.73	0.424	36.33 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	207μs	1.12μs	6.33μs	0.492	0	41.09 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	247μs	955ns	3.7μs	0.497	0	41.7 KB
master	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	311μs	538ns	2.08μs	6.76	0.601	42.89 KB
master	`RunWaf(args=NestedMap (1000))`	net6.0	130μs	110ns	426ns	0.471	0	34.74 KB
master	`RunWaf(args=NestedMap (1000))`	netcoreapp3.1	170μs	613ns	2.38μs	0.414	0	35.39 KB
master	`RunWaf(args=NestedMap (1000))`	net472	213μs	65ns	252ns	5.76	0.427	36.34 KB
master	`RunWafWithAttack(args=Neste(...)tack) [24])`	net6.0	204μs	185ns	718ns	0.52	0	41.09 KB
master	`RunWafWithAttack(args=Neste(...)tack) [24])`	netcoreapp3.1	250μs	975ns	3.78μs	0.488	0	41.69 KB
master	`RunWafWithAttack(args=Neste(...)tack) [24])`	net472	306μs	1.04μs	4.03μs	6.79	0.453	42.9 KB
#4618	`RunWaf(args=NestedMap (10))`	net6.0	52μs	208ns	804ns	0.211	0	16.07 KB
#4618	`RunWaf(args=NestedMap (10))`	netcoreapp3.1	69.3μs	393ns	2.78μs	0.212	0	16.06 KB
#4618	`RunWaf(args=NestedMap (10))`	net472	99.8μs	20.4ns	76.5ns	2.53	0.0953	16.14 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [22])`	net6.0	114μs	39.1ns	146ns	0.285	0	22.42 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [22])`	netcoreapp3.1	133μs	429ns	1.55μs	0.279	0	22.37 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [22])`	net472	168μs	266ns	960ns	3.59	0.167	22.71 KB
#4618	`RunWaf(args=NestedMap (100))`	net6.0	120μs	96.3ns	373ns	0.443	0	34.74 KB
#4618	`RunWaf(args=NestedMap (100))`	netcoreapp3.1	165μs	566ns	2.19μs	0.468	0	35.39 KB
#4618	`RunWaf(args=NestedMap (100))`	net472	224μs	1.15μs	5.12μs	5.7	0.43	36.34 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [23])`	net6.0	199μs	189ns	706ns	0.598	0	41.09 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [23])`	netcoreapp3.1	247μs	502ns	1.95μs	0.497	0	41.69 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [23])`	net472	310μs	1.44μs	5.59μs	6.78	0.602	42.9 KB
#4618	`RunWaf(args=NestedMap (1000))`	net6.0	138μs	109ns	424ns	0.469	0	34.74 KB
#4618	`RunWaf(args=NestedMap (1000))`	netcoreapp3.1	164μs	88.2ns	330ns	0.493	0	35.39 KB
#4618	`RunWaf(args=NestedMap (1000))`	net472	217μs	144ns	556ns	5.77	0.436	36.33 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [24])`	net6.0	202μs	152ns	589ns	0.518	0	41.08 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [24])`	netcoreapp3.1	249μs	913ns	3.54μs	0.499	0	41.7 KB
#4618	`RunWafWithAttack(args=Neste(...)tack) [24])`	net472	304μs	124ns	466ns	6.72	0.458	42.89 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	169μs	88.3ns	342ns	0.254	18.09 KB
master	`SendRequest`	netcoreapp3.1	189μs	327ns	1.27μs	0.189	20.25 KB
master	`SendRequest`	net472	0.000182ns	0.000101ns	0.000391ns	0	0 b
#4618	`SendRequest`	net6.0	168μs	141ns	547ns	0.167	18.09 KB
#4618	`SendRequest`	netcoreapp3.1	191μs	459ns	1.78μs	0.19	20.25 KB
#4618	`SendRequest`	net472	0.000602ns	0.00026ns	0.000971ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #4618

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.46 KB	41.95 KB	488 B	1.18%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	537μs	2.4μs	9.28μs	0.519	0	0	41.46 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	624μs	1.7μs	5.9μs	0.316	0	0	41.77 KB
master	`WriteAndFlushEnrichedTraces`	net472	815μs	3.81μs	17.9μs	8.58	2.45	0.408	53.26 KB
#4618	`WriteAndFlushEnrichedTraces`	net6.0	537μs	215ns	833ns	0.514	0	0	41.95 KB
#4618	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	624μs	1.56μs	6.02μs	0.311	0	0	41.94 KB
#4618	`WriteAndFlushEnrichedTraces`	net472	802μs	3.18μs	12.3μs	8.33	2.65	0.379	53.26 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteNonQuery`	net6.0	1.18μs	0.715ns	2.77ns	0.0106	768 B
master	`ExecuteNonQuery`	netcoreapp3.1	1.35μs	0.616ns	2.39ns	0.0101	768 B
master	`ExecuteNonQuery`	net472	1.63μs	0.372ns	1.44ns	0.116	730 B
#4618	`ExecuteNonQuery`	net6.0	1.11μs	0.478ns	1.85ns	0.0105	768 B
#4618	`ExecuteNonQuery`	netcoreapp3.1	1.38μs	0.747ns	2.79ns	0.0104	768 B
#4618	`ExecuteNonQuery`	net472	1.63μs	1.18ns	4.58ns	0.116	730 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`CallElasticsearch`	net6.0	1.2μs	1.37ns	5.32ns	0.0138	0	992 B
master	`CallElasticsearch`	netcoreapp3.1	1.46μs	0.463ns	1.79ns	0.0131	0	992 B
master	`CallElasticsearch`	net472	2.23μs	0.631ns	2.36ns	0.159	0.00112	1 KB
master	`CallElasticsearchAsync`	net6.0	1.31μs	1.64ns	6.12ns	0.0137	0	968 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.51μs	0.815ns	3.05ns	0.0143	0	1.04 KB
master	`CallElasticsearchAsync`	net472	2.49μs	0.776ns	3ns	0.167	0	1.06 KB
#4618	`CallElasticsearch`	net6.0	1.31μs	1.43ns	5.35ns	0.0138	0	992 B
#4618	`CallElasticsearch`	netcoreapp3.1	1.39μs	0.669ns	2.5ns	0.0132	0	992 B
#4618	`CallElasticsearch`	net472	2.37μs	0.844ns	3.27ns	0.158	0.00118	1 KB
#4618	`CallElasticsearchAsync`	net6.0	1.27μs	0.796ns	3.08ns	0.0133	0	968 B
#4618	`CallElasticsearchAsync`	netcoreapp3.1	1.62μs	3.87ns	15ns	0.0136	0	1.04 KB
#4618	`CallElasticsearchAsync`	net472	2.53μs	0.88ns	3.41ns	0.168	0.00126	1.06 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteAsync`	net6.0	1.34μs	0.695ns	2.6ns	0.0129	0	912 B
master	`ExecuteAsync`	netcoreapp3.1	1.41μs	0.595ns	2.22ns	0.0121	0	912 B
master	`ExecuteAsync`	net472	1.73μs	1.34ns	5.19ns	0.139	0.000862	875 B
#4618	`ExecuteAsync`	net6.0	1.31μs	0.515ns	1.93ns	0.0125	0	912 B
#4618	`ExecuteAsync`	netcoreapp3.1	1.42μs	0.549ns	2.05ns	0.0122	0	912 B
#4618	`ExecuteAsync`	net472	1.65μs	0.676ns	2.53ns	0.139	0.000826	875 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	3.8μs	1.7ns	6.35ns	0.0267	1.94 KB
master	`SendAsync`	netcoreapp3.1	4.33μs	2.17ns	8.39ns	0.0325	2.48 KB
master	`SendAsync`	net472	7.23μs	4.65ns	18ns	0.482	3.05 KB
#4618	`SendAsync`	net6.0	3.95μs	1.91ns	7.41ns	0.0275	1.94 KB
#4618	`SendAsync`	netcoreapp3.1	4.41μs	1.38ns	5.15ns	0.0332	2.48 KB
#4618	`SendAsync`	net472	7.18μs	5.5ns	21.3ns	0.483	3.05 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.34μs	3.19ns	12.4ns	0.0231	1.62 KB
master	`EnrichedLog`	netcoreapp3.1	2.01μs	0.971ns	3.63ns	0.0221	1.62 KB
master	`EnrichedLog`	net472	2.38μs	4.33ns	16.8ns	0.244	1.54 KB
#4618	`EnrichedLog`	net6.0	1.44μs	1.14ns	4.43ns	0.023	1.62 KB
#4618	`EnrichedLog`	netcoreapp3.1	2.1μs	1.98ns	7.66ns	0.0217	1.62 KB
#4618	`EnrichedLog`	net472	2.26μs	1.48ns	5.13ns	0.244	1.54 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	114μs	192ns	742ns	0.0571	0	4.21 KB
master	`EnrichedLog`	netcoreapp3.1	118μs	179ns	669ns	0	0	4.21 KB
master	`EnrichedLog`	net472	150μs	159ns	594ns	0.671	0.224	4.38 KB
#4618	`EnrichedLog`	net6.0	114μs	113ns	421ns	0.0571	0	4.21 KB
#4618	`EnrichedLog`	netcoreapp3.1	119μs	184ns	688ns	0	0	4.21 KB
#4618	`EnrichedLog`	net472	148μs	151ns	583ns	0.665	0.222	4.38 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.96μs	1ns	3.75ns	0.0298	2.18 KB
master	`EnrichedLog`	netcoreapp3.1	3.77μs	1.09ns	4.07ns	0.0282	2.18 KB
master	`EnrichedLog`	net472	4.59μs	2.28ns	8.84ns	0.314	1.99 KB
#4618	`EnrichedLog`	net6.0	2.92μs	1.58ns	6.12ns	0.0303	2.18 KB
#4618	`EnrichedLog`	netcoreapp3.1	3.93μs	1.11ns	4.3ns	0.0295	2.18 KB
#4618	`EnrichedLog`	net472	4.62μs	2.86ns	11.1ns	0.316	1.99 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.24μs	0.844ns	3.16ns	0.0162	1.16 KB
master	`SendReceive`	netcoreapp3.1	1.62μs	2.34ns	8.1ns	0.0155	1.16 KB
master	`SendReceive`	net472	2.12μs	2.2ns	8.53ns	0.185	1.16 KB
#4618	`SendReceive`	net6.0	1.3μs	0.595ns	2.3ns	0.0162	1.16 KB
#4618	`SendReceive`	netcoreapp3.1	1.65μs	1.12ns	4.18ns	0.0155	1.16 KB
#4618	`SendReceive`	net472	2.14μs	0.868ns	3.01ns	0.184	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.62μs	1.77ns	6.86ns	0.0209	1.53 KB
master	`EnrichedLog`	netcoreapp3.1	3.66μs	1.16ns	4.33ns	0.0201	1.58 KB
master	`EnrichedLog`	net472	3.97μs	3.62ns	13.5ns	0.309	1.96 KB
#4618	`EnrichedLog`	net6.0	2.51μs	6.22ns	23.3ns	0.0211	1.53 KB
#4618	`EnrichedLog`	netcoreapp3.1	3.57μs	1.65ns	6.38ns	0.0211	1.58 KB
#4618	`EnrichedLog`	net472	3.99μs	1.19ns	4.62ns	0.309	1.96 KB

Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #4618

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1	1.156	773.38	669.00

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	441ns	0.694ns	2.69ns	0.00751	536 B
master	`StartFinishSpan`	netcoreapp3.1	575ns	0.192ns	0.693ns	0.00721	536 B
master	`StartFinishSpan`	net472	621ns	0.285ns	1.07ns	0.0852	538 B
master	`StartFinishScope`	net6.0	551ns	0.176ns	0.657ns	0.00916	656 B
master	`StartFinishScope`	netcoreapp3.1	773ns	0.459ns	1.78ns	0.00888	656 B
master	`StartFinishScope`	net472	847ns	0.321ns	1.24ns	0.098	618 B
#4618	`StartFinishSpan`	net6.0	441ns	0.176ns	0.682ns	0.00746	536 B
#4618	`StartFinishSpan`	netcoreapp3.1	617ns	0.217ns	0.814ns	0.00709	536 B
#4618	`StartFinishSpan`	net472	603ns	0.311ns	1.12ns	0.0853	538 B
#4618	`StartFinishScope`	net6.0	542ns	0.204ns	0.79ns	0.00927	656 B
#4618	`StartFinishScope`	netcoreapp3.1	669ns	0.288ns	1.12ns	0.00868	656 B
#4618	`StartFinishScope`	net472	835ns	0.127ns	0.493ns	0.098	618 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	563ns	0.244ns	0.946ns	0.00937	656 B
master	`RunOnMethodBegin`	netcoreapp3.1	811ns	0.538ns	2.01ns	0.00888	656 B
master	`RunOnMethodBegin`	net472	887ns	0.283ns	1.02ns	0.098	618 B
#4618	`RunOnMethodBegin`	net6.0	537ns	0.162ns	0.608ns	0.00906	656 B
#4618	`RunOnMethodBegin`	netcoreapp3.1	823ns	0.451ns	1.75ns	0.00907	656 B
#4618	`RunOnMethodBegin`	net472	943ns	0.283ns	1.09ns	0.0982	618 B

andrewlock · 2023-09-13T10:32:30Z

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4618) (11.424M)   : 0, 11423634
    master (11.785M)   : 0, 11784896
    benchmarks/2.37.0 (11.517M)   : 0, 11517370
    benchmarks/2.9.0 (11.506M)   : 0, 11506345

    section Automatic
    This PR (4618) (8.085M)   : 0, 8085218
    master (8.157M)   : 0, 8156884
    benchmarks/2.37.0 (8.143M)   : 0, 8142506
    benchmarks/2.9.0 (8.280M)   : 0, 8279564

    section Trace stats
    master (8.126M)   : 0, 8125619
    benchmarks/2.37.0 (8.044M)   : 0, 8044129

    section Manual
    This PR (4618) (10.122M)   : 0, 10121906
    master (10.297M)   : 0, 10296663
    benchmarks/2.37.0 (10.278M)   : 0, 10278226

    section Manual + Automatic
    This PR (4618) (7.754M)   : 0, 7753625
    master (7.837M)   : 0, 7836527
    benchmarks/2.37.0 (7.786M)   : 0, 7785678

    section Version Conflict
    master (7.047M)   : 0, 7047317
    benchmarks/2.37.0 (7.067M)   : 0, 7066760

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4618) (9.414M)   : 0, 9413657
    master (9.607M)   : 0, 9606862
    benchmarks/2.37.0 (9.619M)   : 0, 9618848
    benchmarks/2.9.0 (9.542M)   : 0, 9542061

    section Automatic
    This PR (4618) (6.811M)   : 0, 6811351
    master (6.552M)   : 0, 6552425
    benchmarks/2.37.0 (6.844M)   : 0, 6843850

    section Trace stats
    master (6.757M)   : 0, 6757190
    benchmarks/2.37.0 (6.801M)   : 0, 6801072

    section Manual
    This PR (4618) (8.627M)   : 0, 8627441
    master (8.482M)   : 0, 8482481
    benchmarks/2.37.0 (8.587M)   : 0, 8587283

    section Manual + Automatic
    This PR (4618) (6.486M)   : 0, 6485653
    master (6.516M)   : 0, 6516148
    benchmarks/2.37.0 (6.537M)   : 0, 6536939

    section Version Conflict
    master (5.876M)   : 0, 5875591
    benchmarks/2.37.0 (5.769M)   : 0, 5768881

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4618) (9.281M)   : 0, 9281252
    master (9.292M)   : 0, 9292072
    benchmarks/2.37.0 (9.309M)   : 0, 9308797
    benchmarks/2.9.0 (9.366M)   : 0, 9366476

    section Automatic
    This PR (4618) (6.668M)   : 0, 6667939
    master (6.776M)   : 0, 6775885
    benchmarks/2.37.0 (6.552M)   : 0, 6552099
    benchmarks/2.9.0 (6.864M)   : 0, 6864235

    section Trace stats
    master (6.734M)   : 0, 6734320
    benchmarks/2.37.0 (6.685M)   : 0, 6685340

    section Manual
    This PR (4618) (8.354M)   : 0, 8354376
    master (8.423M)   : 0, 8423266
    benchmarks/2.37.0 (8.493M)   : 0, 8492505

    section Manual + Automatic
    This PR (4618) (6.538M)   : 0, 6537776
    master (6.451M)   : 0, 6451144
    benchmarks/2.37.0 (6.373M)   : 0, 6372588

    section Version Conflict
    master (5.974M)   : 0, 5974442
    benchmarks/2.37.0 (5.938M)   : 0, 5937795

gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4618) (7.355M)   : 0, 7355227
    master (7.253M)   : 0, 7253384
    benchmarks/2.37.0 (7.239M)   : 0, 7239363
    benchmarks/2.9.0 (7.508M)   : 0, 7508023

    section No attack
    This PR (4618) (2.078M)   : 0, 2078183
    master (2.036M)   : 0, 2035944
    benchmarks/2.37.0 (2.105M)   : 0, 2105389
    benchmarks/2.9.0 (3.275M)   : 0, 3275098

    section Attack
    This PR (4618) (1.627M)   : 0, 1627091
    master (1.660M)   : 0, 1659904
    benchmarks/2.37.0 (1.686M)   : 0, 1685550
    benchmarks/2.9.0 (2.570M)   : 0, 2570112

    section Blocking
    This PR (4618) (3.169M)   : 0, 3169192
    master (3.228M)   : 0, 3227614
    benchmarks/2.37.0 (3.048M)   : 0, 3048473

tonyredondo

LGTM, just a nit: I think we can add the InitTime to CIVisibility mode

tonyredondo · 2023-09-14T13:26:25Z

...metryMetric.TelemetryMetricGenerator/CiVisibilityMetricsTelemetryCollector_distribution.g.cs

+    public void RecordDistributionInitTime(Datadog.Trace.Telemetry.Metrics.MetricTags.InitializationComponent tag, double value)
+    {
+    }


I think InitTime could be useful in a CI Visibility scenario...

andrewlock added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) area:ci-visibility area:telemetry labels Sep 13, 2023

andrewlock requested review from a team as code owners September 13, 2023 09:40

andrewlock force-pushed the andrew/telemetry/ci-app-metrics branch from 98335ea to 0b8355c Compare September 13, 2023 16:23

tonyredondo approved these changes Sep 14, 2023

View reviewed changes

pierotibou approved these changes Sep 14, 2023

View reviewed changes

andrewlock mentioned this pull request Sep 14, 2023

[CI Visibility] Telemetry metrics #4596

Merged

andrewlock force-pushed the andrew/telemetry/ci-app-metrics branch from 0b8355c to 5032cf4 Compare September 15, 2023 11:02

andrewlock added 7 commits September 15, 2023 13:31

Extract base class from MetricsTelemetryCollector

42c5915

Refactor: extract the method generation

fd237a8

Add additional CiVisibility Metrics TelemetryCollector

bc3598c

Add explicit ciapp test

e8b2d79

Allow using the alternative metrics controller in ci-app

7f4f7d3

Add hack for the fact that _buffer is accessed in derived types now

72c319e

Add source generator output

c1bf698

andrewlock force-pushed the andrew/telemetry/ci-app-metrics branch from 5032cf4 to c1bf698 Compare September 15, 2023 12:31

andrewlock merged commit de00390 into master Sep 15, 2023
49 of 51 checks passed

andrewlock deleted the andrew/telemetry/ci-app-metrics branch September 15, 2023 13:40

github-actions bot added this to the vNext milestone Sep 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restrict sending APM-related metrics when running in CI app #4618

Restrict sending APM-related metrics when running in CI app #4618

andrewlock commented Sep 13, 2023

datadog-ddstaging bot commented Sep 13, 2023 •

edited

Loading

andrewlock commented Sep 13, 2023 •

edited

Loading

andrewlock commented Sep 13, 2023 •

edited

Loading

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #4618

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4618

Raw results

Raw results

andrewlock commented Sep 13, 2023 •

edited

Loading

tonyredondo left a comment

tonyredondo Sep 14, 2023

Restrict sending APM-related metrics when running in CI app #4618

Restrict sending APM-related metrics when running in CI app #4618

Conversation

andrewlock commented Sep 13, 2023

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

datadog-ddstaging bot commented Sep 13, 2023 • edited Loading

Datadog Report

andrewlock commented Sep 13, 2023 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Sep 13, 2023 • edited Loading

Benchmarks Report 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #4618

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Faster 🎉 in #4618

Raw results

Raw results

andrewlock commented Sep 13, 2023 • edited Loading

Throughput/Crank Report:zap:

tonyredondo left a comment

Choose a reason for hiding this comment

tonyredondo Sep 14, 2023

Choose a reason for hiding this comment

datadog-ddstaging bot commented Sep 13, 2023 •

edited

Loading

andrewlock commented Sep 13, 2023 •

edited

Loading

andrewlock commented Sep 13, 2023 •

edited

Loading

andrewlock commented Sep 13, 2023 •

edited

Loading