chore(o11y): split internal telemetry across INFO/DEBUG/TRACE and set default level to INFO

[tobz]

Summary

This PR introduces a first pass at starting to classify Saluki's internal telemetry at more appropriate levels to allow for effectively filtering out useful, always-on telemetry from more low level/granular telemetry that is primarily useful for debugging.

Saluki, based on the underlying metrics library it uses, has a concept of metric "levels": levels are synonymous with the levels you see in logging (INFO, DEBUG, etc), and are designed to allow for filtering out expensive/high-cardinality metrics that might not always be necessary. Most of the internal telemetry emitted by Saluki was already classified at different levels, but the leveling wasn't consistent across the board, and we had a default level filter of TRACE, meaning we always emitted everything anyways.

In this PR, we've done a few things:

• added the ability to configure a default metric filter level, and set that default to INFO
• reclassified the level of many internal metrics to reduce the volume when filtering at the INFO level
• fixed a bug with histogram sample accumulation when a histogram is actively filtered (Histograms in internal telemetry can accumulate forever if actively filtered. #1534)

Level reclassification

We took a simple but principled approach during reclassification:

• metrics that are ultimately used for RAR/COAT should be INFO so we never skip emitting them
• component telemetry (almost everything that looks like component_*_total) should be INFO as this provides general performance/health indicators for the topology
• almost all other things should be DEBUG
• anything in the DEBUG category which is very high-cardinality (some runtime metrics, task poll latency, etc) becomes TRACE just to give a little bit of separation

One additional change is that we removed a remapped metrics -- "dogstatsd.channel_latency_secs -- ads it would have required us to make a high-cardinality histogram sit at the INFO level, and we don't actually need to mirror that metric because it was always sort of an approximation of the original metric to begin with.

Default INFO level

With the metric filter level now exposed in configuration (metrics_level, or DD_METRICS_LEVEL), we can now trivially update the default filter level to INFO as it can be overridden from the default in scenarios where a user always wants DEBUG or TRACE metrics flowing. Additionally, we already have existing support for dynamically adjusting the metrics level via agent-data-plane debug set-metric-level.

Required follow-up

This PR constrains the emission of internal telemetry, but one thing it doesn't yet change is how the internal observability topology holds on to it. We currently use a Prometheus destination for exposing our internal telemetry outside of the process, and a "reflector" for holding it prior to sending it over RAR. Both of these mechanisms will accumulate whatever metrics they receive without any additional filtering or clearing out of metrics once the metrics filter level changes.

Essentially, this means that once the metrics filter level is changed to something more verbose, that "level" will be exposed until the process is restarted even if the more-verbose metrics aren't actually being updated.

We'll need to make some subsequent changes to the aforementioned mechanisms so that they can participate and observe the changes to the metric levels (somehow), allowing them to clear out metrics which fall outside of the updated metrics filter level to stop them from being exposed.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

• Existing unit tests.
• Ran ADP (standalone mode) and observed only INFO level metrics in internal telemetry output.
• Ran ADP (standalone mode) with metrics_level: trace and observed all metrics in internal telemetry output.

References

DADP-65

Fixes #1534

[pr-commenter]

Regression Detector (Agent Data Plane)

Regression Detector Results

Run ID: dd94e203-7a49-4cdc-afe0-14a16fff03bd

Baseline: d04bde8
Comparison: 3c050b0
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	otlp_ingest_logs_5mb_cpu	% cpu utilization	+1.25	[-3.05, +5.56]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_logs_5mb_throughput	ingress throughput	-0.01	[-0.13, +0.11]	1	(metrics) (profiles) (logs)
✅	otlp_ingest_logs_5mb_memory	memory utilization	-8.24	[-8.65, -7.84]	1	(metrics) (profiles) (logs)

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	dsd_uds_500mb_3k_contexts_throughput	ingress throughput	+2.90	[+2.77, +3.03]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_logs_5mb_cpu	% cpu utilization	+1.25	[-3.05, +5.56]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_metrics_5mb_cpu	% cpu utilization	+0.92	[-5.13, +6.96]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_5mb_cpu	% cpu utilization	+0.81	[-1.38, +3.00]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_5mb_throughput	ingress throughput	+0.23	[+0.16, +0.30]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_filtering_5mb_throughput	ingress throughput	+0.22	[+0.15, +0.30]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_transform_5mb_throughput	ingress throughput	+0.19	[+0.11, +0.26]	1	(metrics) (profiles) (logs)
➖	dsd_uds_512kb_3k_contexts_throughput	ingress throughput	+0.01	[-0.05, +0.06]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_transform_5mb_cpu	% cpu utilization	+0.01	[-2.01, +2.02]	1	(metrics) (profiles) (logs)
➖	dsd_uds_1mb_3k_contexts_throughput	ingress throughput	-0.00	[-0.06, +0.06]	1	(metrics) (profiles) (logs)
➖	dsd_uds_100mb_3k_contexts_throughput	ingress throughput	-0.00	[-0.04, +0.03]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_metrics_5mb_throughput	ingress throughput	-0.01	[-0.15, +0.14]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_logs_5mb_throughput	ingress throughput	-0.01	[-0.13, +0.11]	1	(metrics) (profiles) (logs)
➖	dsd_uds_10mb_3k_contexts_throughput	ingress throughput	-0.01	[-0.19, +0.16]	1	(metrics) (profiles) (logs)
➖	dsd_uds_500mb_3k_contexts_cpu	% cpu utilization	-0.69	[-2.12, +0.73]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_metrics_5mb_memory	memory utilization	-0.70	[-0.89, -0.52]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_filtering_5mb_cpu	% cpu utilization	-0.84	[-3.33, +1.65]	1	(metrics) (profiles) (logs)
➖	quality_gates_rss_dsd_ultraheavy	memory utilization	-0.87	[-1.01, -0.74]	1	(metrics) (profiles) (logs)
➖	quality_gates_rss_dsd_heavy	memory utilization	-1.20	[-1.32, -1.07]	1	(metrics) (profiles) (logs)
➖	dsd_uds_100mb_3k_contexts_cpu	% cpu utilization	-1.22	[-6.98, +4.54]	1	(metrics) (profiles) (logs)
➖	dsd_uds_1mb_3k_contexts_cpu	% cpu utilization	-1.75	[-56.10, +52.61]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_transform_5mb_memory	memory utilization	-1.95	[-2.11, -1.79]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_5mb_memory	memory utilization	-2.12	[-2.27, -1.96]	1	(metrics) (profiles) (logs)
➖	otlp_ingest_traces_ottl_filtering_5mb_memory	memory utilization	-2.35	[-2.59, -2.11]	1	(metrics) (profiles) (logs)
➖	dsd_uds_512kb_3k_contexts_cpu	% cpu utilization	-2.57	[-58.99, +53.84]	1	(metrics) (profiles) (logs)
➖	quality_gates_rss_dsd_medium	memory utilization	-2.64	[-2.80, -2.47]	1	(metrics) (profiles) (logs)
➖	dsd_uds_10mb_3k_contexts_memory	memory utilization	-3.50	[-3.65, -3.35]	1	(metrics) (profiles) (logs)
➖	dsd_uds_100mb_3k_contexts_memory	memory utilization	-3.59	[-3.74, -3.44]	1	(metrics) (profiles) (logs)
➖	dsd_uds_1mb_3k_contexts_memory	memory utilization	-3.61	[-3.75, -3.48]	1	(metrics) (profiles) (logs)
➖	quality_gates_rss_dsd_low	memory utilization	-3.74	[-3.89, -3.58]	1	(metrics) (profiles) (logs)
➖	dsd_uds_512kb_3k_contexts_memory	memory utilization	-3.96	[-4.10, -3.82]	1	(metrics) (profiles) (logs)
➖	dsd_uds_500mb_3k_contexts_memory	memory utilization	-3.97	[-4.10, -3.83]	1	(metrics) (profiles) (logs)
➖	quality_gates_rss_idle	memory utilization	-4.42	[-4.46, -4.38]	1	(metrics) (profiles) (logs)
➖	dsd_uds_10mb_3k_contexts_cpu	% cpu utilization	-5.72	[-35.77, +24.33]	1	(metrics) (profiles) (logs)
✅	otlp_ingest_logs_5mb_memory	memory utilization	-8.24	[-8.65, -7.84]	1	(metrics) (profiles) (logs)

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	quality_gates_rss_dsd_heavy	memory_usage	10/10	123.04MiB ≤ 140MiB	(metrics) (profiles) (logs)
✅	quality_gates_rss_dsd_low	memory_usage	10/10	39.50MiB ≤ 50MiB	(metrics) (profiles) (logs)
✅	quality_gates_rss_dsd_medium	memory_usage	10/10	60.15MiB ≤ 75MiB	(metrics) (profiles) (logs)
✅	quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	176.40MiB ≤ 200MiB	(metrics) (profiles) (logs)
✅	quality_gates_rss_idle	memory_usage	10/10	27.02MiB ≤ 40MiB	(metrics) (profiles) (logs)

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[tobz]

These were just missed updates to the list of title scopes.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 02de2fb605

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Target: d04bde8 (baseline) vs 3c050b0 (comparison) diff
Analysis Type: Stripped binaries (debug symbols excluded)
Baseline Size: 37.55 MiB
Comparison Size: 37.54 MiB
Size Change: -8.22 KiB (-0.02%)
Pass/Fail Threshold: +5%
Result: PASSED ✅

Changes by Module

Module	File Size	Symbols
prost	+11.40 KiB	24
alloc	-9.59 KiB	69
rustls	-9.13 KiB	10
tokio_rustls	+9.02 KiB	8
tower	-5.73 KiB	19
figment	-5.32 KiB	40
saluki_io::net::util	+4.96 KiB	4
saluki_config::provider::ResolvedProvider	+4.61 KiB	1
tonic	+4.05 KiB	41
tonic_prost	-4.00 KiB	16
chrono	-3.99 KiB	6
anon.881a6e9513295f4e2155425a0393b194.103.llvm.17973815463175268679	+3.55 KiB	1
anon.f3ef97b7c762ee5da43eb1ee76d1e062.10.llvm.91222540921911643	-3.55 KiB	1
saluki_config::ConfigurationLoader::from_yaml	-3.50 KiB	1
tokio_util	-3.16 KiB	13
anon.37efd500d662fac30494dfcb6655a0cd.162.llvm.15594617127947676363	+3.06 KiB	1
anon.c1c7ec8ec680e4785be08d066b5ec12c.2.llvm.657345546224115985	-3.06 KiB	1
serde_yaml	-2.84 KiB	18
saluki_context::resolver::ContextResolver	-2.73 KiB	1
[Unmapped]	-2.72 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +523% +9.12Ki  +549% +9.03Ki    tokio_rustls::common::Stream<IO,C>::read_io::h79398d61ce27274c
  [NEW] +4.85Ki  [NEW] +4.66Ki    _<saluki_io::net::util::middleware::retry_circuit_breaker::ResponseFuture<P,F,Request> as core::future::future::Future>::poll::h1e36675bd5cac502
  [NEW] +4.61Ki  [NEW] +4.51Ki    saluki_config::provider::ResolvedProvider::from_yaml::he5a7124d3a4f3b15
  [NEW] +4.30Ki  [NEW] +4.12Ki    _<saluki_io::deser::framing::NestedFramer<Inner,Outer> as saluki_io::deser::framing::Framer>::next_frame::h0e8a40ccd8272d1e
  [NEW] +3.96Ki  [NEW]    +458    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_app::metrics::api::MetricsHandlerState>>,axum::routing::path_router::PathRouter<saluki_app::metrics::api::MetricsHandlerState,_>::with_state<$LP$$RP$>::{{closure}}>>::h9ed5c14b830c5510
  [NEW] +3.55Ki  [NEW]     +16    anon.881a6e9513295f4e2155425a0393b194.103.llvm.17973815463175268679
  [NEW] +3.37Ki  [NEW] +3.15Ki    serde_json::value::de::_<impl serde_core::de::Deserializer for serde_json::map::Map<alloc::string::String,serde_json::value::Value>>::deserialize_any::had44c9609cb7dfe9
   +43% +3.36Ki   +45% +3.36Ki    _<std::io::error::ErrorKind as core::fmt::Debug>::fmt::h6f31d7a057898f75
  [NEW] +3.06Ki  [NEW]     +74    anon.37efd500d662fac30494dfcb6655a0cd.162.llvm.15594617127947676363
  [NEW] +2.90Ki  [NEW]    +329    core::ptr::drop_in_place<sync_wrapper::SyncWrapper<alloc::boxed::Box<dyn tonic::codec::Decoder+Item = datadog_protos::agent_include::datadog::model::v1::HostTagReply+Error = tonic::status::Status+core::marker::Send>>>::h64ee8db19078c942
  [DEL] -3.06Ki  [DEL]     -74    anon.c1c7ec8ec680e4785be08d066b5ec12c.2.llvm.657345546224115985
 -20.6% -3.18Ki -20.8% -3.18Ki    _<chrono::format::formatting::DelayedFormat<I> as core::fmt::Display>::fmt::hde220a98556cc7ac
 -80.8% -3.50Ki -82.6% -3.50Ki    saluki_config::ConfigurationLoader::from_yaml::hc12067095c71b6b2
  [DEL] -3.55Ki  [DEL]     -16    anon.f3ef97b7c762ee5da43eb1ee76d1e062.10.llvm.91222540921911643
  [DEL] -3.96Ki  [DEL]    -458    core::ptr::drop_in_place<core::iter::adapters::map::Map<std::collections::hash::map::IntoIter<axum::routing::RouteId,axum::routing::Endpoint<saluki_app::config::ConfigState>>,axum::routing::path_router::PathRouter<saluki_app::config::ConfigState,_>::with_state<$LP$$RP$>::{{closure}}>>::h6934bcec16b3c802
  [DEL] -4.32Ki  [DEL] -4.16Ki    _<saluki_io::deser::framing::Framed<F,B> as core::iter::traits::iterator::Iterator>::next::ha945f7631bececf2
 -92.4% -4.58Ki -95.7% -4.58Ki    serde_json::value::de::_<impl serde_core::de::Deserializer for serde_json::value::Value>::deserialize_struct::h1ddb34607fa53842
  [DEL] -4.90Ki  [DEL] -4.74Ki    _<tower::limit::concurrency::future::ResponseFuture<F> as core::future::future::Future>::poll::h4ed01fa9a8bc2dc5
 -52.6% -5.19Ki -53.5% -5.19Ki    _<alloc::collections::btree::map::BTreeMap<K,V> as figment::coalesce::Coalescible>::coalesce::hb47602d17e1065f4
  -0.1% -6.44Ki  -0.2% -8.57Ki    [8063 Others]
  [DEL] -8.62Ki  [DEL] -8.51Ki    rustls::conn::ConnectionCore<Data>::process_new_packets::h206c129c8510e48d
  -0.0% -8.22Ki  -0.0% -13.3Ki    TOTAL