chore(config): migrate Datadog Logs to typed config

[webern]

Human Summary

TODO: human writes here

AI Summary

Build the Datadog Logs encoder from the shared typed compression configuration and remove its raw configuration deserialization.

Change Type

• Non-functional (chore, refactoring, docs)

How did you test this PR?

• make build-schema-overlay && make fmt
• make check-all
• make test
• make check-docs

References

[Copilot]

Pull request overview

This PR migrates the Datadog Logs encoder configuration away from raw GenericConfiguration deserialization and instead builds it from the shared typed compression configuration exposed by agent_data_plane_config.

Changes:

• Replaced DatadogLogsConfiguration’s serde-based config deserialization with a constructor that maps from agent_data_plane_config::shared::Compression.
• Updated the agent-data-plane topology wiring to pass saluki.shared.endpoints.compression into the logs pipeline builder.
• Replaced the config smoke test with a unit test validating the new mapping behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
lib/saluki-components/src/encoders/datadog/logs/mod.rs	Switch logs encoder config to be constructed from shared typed compression settings; adjust tests accordingly.
bin/agent-data-plane/src/cli/run.rs	Wire the logs pipeline to use typed shared compression config instead of generic config.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 38a66a87c9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve ADP's zstd default for logs

When serializer_zstd_compressor_level is omitted, this now copies the shared typed value populated from the generated Datadog config default of 1 (lib/datadog-agent/config/src/generated/datadog_configuration.rs:354), but the overlay explicitly documents that ADP intentionally defaults this key to level 3 while the Agent defaults to 1, and the removed logs config used default_zstd_compressor_level() -> 3. Default log pipelines therefore silently switch to weaker zstd compression while the other encoders still use level 3, increasing log payload sizes for users who do not set the key.

Useful? React with 👍 / 👎.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 7b98c6f · Comparison: aa34a87 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.63 MiB (baseline) vs 42.20 MiB (comparison)
Size Change: +579.12 KiB (+1.36%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
figment	-300.64 KiB	433
serde_json	+232.94 KiB	506
core	+161.95 KiB	9324
datadog_agent_config::generated::datadog_configuration	+118.77 KiB	55
serde_core	+65.59 KiB	938
saluki_components::common::datadog	+53.93 KiB	571
alloc	+51.65 KiB	1456
prost	+37.75 KiB	259
tokio	+36.08 KiB	3083
[sections]	+32.00 KiB	8
serde	-30.41 KiB	86
agent_data_plane_config_system::translators::datadog_translator	+26.46 KiB	22
tracing	-20.36 KiB	115
otlp_protos::otlp_include::opentelemetry	-20.06 KiB	153
axum	+18.06 KiB	356
serde_with	-15.90 KiB	46
piecemeal	+15.04 KiB	43
hyper	+14.02 KiB	265
datadog_protos::trace_piecemeal_include::datadog	-13.82 KiB	25
datadog_agent_config::generated::witness	+12.38 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +2.9%  +462Ki  +2.4%  +288Ki    [33927 Others]
  [NEW] +55.2Ki  [NEW] +55.0Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h3ef2a484186f0bb8
  [NEW] +53.3Ki  [NEW] +52.9Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h59abc0a47db21533
  [NEW] +53.2Ki  [NEW] +53.1Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::he5819430e26508ec
  [NEW] +39.2Ki  [NEW] +39.0Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h3d7ef516f362b9a8
  [NEW] +36.9Ki  [NEW] +36.8Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::hd269a5b16c8f1266
  [NEW] +36.4Ki  [NEW] +36.1Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h5dc7893c0984844c
  [NEW] +32.1Ki  [NEW] +31.9Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h16acca5cc1c6ba60
  [NEW] +31.9Ki  [NEW] +31.7Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h65e15c9238557bac
  [NEW] +31.7Ki  [NEW] +31.5Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h6809d072204e0804
  [NEW] +29.8Ki  [NEW] +29.6Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hd22ba2289c2b0af8
  [NEW] +26.4Ki  [NEW] +26.1Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hba6844531adf2ec6
  [DEL] -25.6Ki  [DEL] -25.4Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h3fe2520e3abdfc38
  [DEL] -26.6Ki  [DEL] -26.4Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::h2c2a33d551977c6d
  [DEL] -27.2Ki  [DEL] -26.9Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hfc578ba92a99992a
  [DEL] -28.6Ki  [DEL] -28.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h1e8351a6f731e9df
  [DEL] -28.6Ki  [DEL] -28.5Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h84db3f7ad161eab2
  [DEL] -32.6Ki  [DEL] -32.4Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h72e4b09fd816d94f
  [DEL] -38.7Ki  [DEL] -38.5Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h3cd6ee67f0471508
  [DEL] -44.7Ki  [DEL] -44.6Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h83501a7881036c5f
  [DEL] -56.8Ki  [DEL] -56.6Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h54fbb73bb30184e6
  +1.4%  +579Ki  +1.2%  +404Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 14d0ccd6-b8be-499c-88fd-de4fe669702b
Baseline: 7b98c6f1 · Comparison: 38a66a87 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (3)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_idle	memory	⚪ +1.95	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +1.67	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +1.03	metrics profiles logs

Bounds Checks: ✅ Passed (3)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 43.6 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 65.5 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 29.5 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[webern]

Review by Claude Code, operating on behalf of @webern.

Summary

The mechanical cutover is clean: the encoder now builds from shared.endpoints.compression, the two keys (serializer_compressor_kind, serializer_zstd_compressor_level) remain reachable through the schema → translator path, and no input validation/coercion was dropped (nothing existed at the component layer before; the values still flow straight into CompressionScheme::new). CI, regression detector, and bounds checks are all green.

There is one real regression, however, which Copilot and Codex both also caught.

Default zstd compressor level regresses 3 → 1

The removed component code defaulted serializer_zstd_compressor_level to 3:

const fn default_zstd_compressor_level() -> i32 { 3 }

After this PR the value comes from shared.endpoints.compression.zstd_compressor_level, which the translator populates from the generated Datadog config default of 1 (lib/datadog-agent/config/src/generated/datadog_configuration.rs, serializer_zstd_compressor_level: defaults::default_u64::<i64, 1>()). Nothing seeds an ADP default of 3 at the shared/typed layer, so when the key is unset the effective default silently drops from 3 to 1.

This directly contradicts the documented intent in schema_overlay.yaml:

"ADP defaults to level 3 vs Agent level 1. ADP intentionally compresses better. Key is recognized by both; default divergence is intentional."

The overlay documents the divergence in prose but never actually sets default: 3, so the generated default is 1. Net effect: default log pipelines get weaker zstd compression (larger payloads), and Logs now diverges from the sibling Datadog encoders (metrics/events/service-checks/traces) that still default to 3 until they're migrated.

Suggested fix

Per the "defaults live in the config layer" principle, establish ADP's intended default of 3 at the typed/shared layer rather than in the component — e.g. give serializer_zstd_compressor_level a default: 3 override in schema_overlay.yaml so the generated default matches the documented intent. Doing it at the shared layer fixes it uniformly for every Datadog encoder that reads shared.endpoints.compression, not just Logs.

serializer_compressor_kind is unaffected — both the old component default and the schema default are zstd.

Tests

Dropping the old config_smoke test is fine — it was purely a deserialization/remapping check. The replacement only verifies a trivial field copy, so nothing now covers the effective default value. A translator-level assertion that shared.endpoints.compression.zstd_compressor_level defaults to the intended value would have caught this regression and is worth adding alongside the fix.

[webern]

Follow-up by Claude Code, operating on behalf of @webern.

The zstd-default regression I flagged earlier is resolved by the rebase. The fix now lives in the config layer, where it belongs:

• New constant ZSTD_DEFAULT_OVERRIDE = 3 in agent-data-plane-config/src/shared.rs.
• consume_serializer_zstd_compressor_level compares the incoming level to the Agent's schema default and substitutes ADP's intended level 3 when they match, leaving any explicit operator value untouched.

shared.endpoints.compression.zstd_compressor_level now defaults to 3, matching the documented ADP intent and the old component behavior, and it applies uniformly to every Datadog encoder rather than just Logs. The trivial 1→3 substitution doesn't warrant a dedicated test.

The documented caveat stands — an operator who explicitly sets level 1 is indistinguishable from the Agent default and is also bumped to 3 — but that's noted in-code as follow-up work.

No remaining concerns. This is ready to merge.