Skip to content

chore(config): build the first components from the typed configuration model#1979

Draft
webern wants to merge 2 commits into
m/pr3b-env-keysfrom
m/pr4-wired
Draft

chore(config): build the first components from the typed configuration model#1979
webern wants to merge 2 commits into
m/pr3b-env-keysfrom
m/pr4-wired

Conversation

@webern

@webern webern commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Human Summary

This PR serves as a foundational model for cutting over components to typed config. It serves two purposes:

  • Show human reviewers what it looks like to cut over two configurations to the typed model in a controlled diff.
  • Serves as a reference for coding agents performing the rest of the cutovers.

An issue with dotted path environment variable support also had to be dealt with in this PR rather than laboriously shuttling it downward in the stack.

Key aliases that seemed to have arisen by mistake, such that we have a "saluki only" key for something that is Datadog schema-provided are treated as bugs as we go and the "saluki-only" aliases will be removed. I believe there are only a couple cases of this, but one of them happens to be in this PR.

AI Summary

Establishes the reference pattern for constructing pipeline components from the typed
SalukiConfiguration model instead of the raw GenericConfiguration map, and cuts two DogStatsD
components over to it.

  • Aggregate transform (static): AggregateConfiguration builds from the typed DogStatsD
    aggregation slice and shared histogram-encoding settings. Source serde behavior is removed from
    the component. Histogram parsing moves to a fallible conversion, and construction rejects a zero
    aggregation window.
  • Debug-log destination (dynamic): DogStatsDDebugLogConfiguration builds from the typed
    debug-log slice and holds a Live<DebugLog>. The destination reacts to runtime changes through
    the typed live view instead of watching a raw string key.

Topology assembly now supplies each component its typed configuration subtree and, where needed, a
live view from ConfigurationSystem.

The aggregate cutover also fixes two duplicate configuration paths:

  • dogstatsd_flush_incomplete_buckets now drives the aggregation field the transform reads; the
    duplicate Saluki-only path is removed.
  • dogstatsd_expiry_seconds is the sole source for counter expiry; the duplicate Saluki-only key is
    removed.

Translation seeds Saluki-only values first and applies the Datadog source afterward, making the
Datadog source authoritative for every field it owns. Obsolete component-level configuration smoke
tests are removed; equivalent coverage lives in translation and component construction tests.

Change Type

  • Non-functional (chore, refactoring, docs)

How did you test this PR?

  • cargo test for saluki-components, agent-data-plane-config, and
    agent-data-plane-config-system
  • cargo check -p agent-data-plane
  • Clippy on the touched crates
  • Regenerated the schema overlay, configuration registry, and documentation

Integration and correctness suites were not run locally.

References

@dd-octo-sts dd-octo-sts Bot added area/components Sources, transforms, and destinations. transform/aggregate Aggregate transform. area/docs Reference documentation. labels Jul 4, 2026
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jul 4, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

DataDog/saluki | test-integration   View in Datadog   GitLab

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0a7a0e6 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 4, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 2715682 · Comparison: 0a7a0e6 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.54 MiB (baseline) vs 42.31 MiB (comparison)
Size Change: +789.50 KiB (+1.86%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
serde_json +241.45 KiB 506
core +185.73 KiB 9241
figment -166.18 KiB 474
datadog_agent_config::generated::datadog_configuration +113.59 KiB 54
serde_core +70.58 KiB 932
agent_data_plane_config_system::saluki_env_overlay::PathRecorder +65.96 KiB 25
tracing -46.62 KiB 122
serde -44.62 KiB 85
saluki_components::common::datadog +42.55 KiB 560
hyper_util +35.67 KiB 79
alloc +34.80 KiB 1709
serde_with +21.69 KiB 56
[sections] +21.32 KiB 8
agent_data_plane_config_system::translators::datadog_translator +21.15 KiB 21
anon.07d452140bcd0886c53588209cedc189.105.llvm.16230740086584210351 +17.77 KiB 1
anon.5f040b9cf0e85edcd98481965dce5e0f.55.llvm.9953283433741861364 -17.07 KiB 1
hashbrown +16.54 KiB 863
tonic -15.19 KiB 411
&mut serde_json -14.48 KiB 94
piecemeal +14.36 KiB 48
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +3.7%  +679Ki  +3.3%  +478Ki    [35567 Others]
  [NEW] +58.7Ki  [NEW] +58.5Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h111b2001394d1b43
  [NEW] +53.5Ki  [NEW] +53.2Ki    _<datadog_agent_config::generated::datadog_configuration::_::<impl serde_core::de::Deserialize for datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hb8fedc8e9123324e
  [NEW] +44.5Ki  [NEW] +44.4Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::haf7f24f8b653b2ef
  [NEW] +41.4Ki  [NEW] +41.2Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h0c3aaf17069cc10f
  [NEW] +40.8Ki  [NEW] +40.6Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h8d0275d23b6b11d4
  [NEW] +36.3Ki  [NEW] +36.0Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h723ca7e38e02760c
  [NEW] +31.8Ki  [NEW] +31.6Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h0e662b74c7a03705
  [NEW] +30.1Ki  [NEW] +30.0Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hd9dc7e3e44cb977e
  [NEW] +30.0Ki  [NEW] +29.9Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h5f2e67de25b0e1ba
  [NEW] +28.4Ki  [NEW] +28.3Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hdf97b4715cf33261
  [NEW] +28.4Ki  [NEW] +28.2Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h6d26598a9f09a68f
  [DEL] -27.4Ki  [DEL] -27.1Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hf7c926a0582af9a6
  [DEL] -28.0Ki  [DEL] -27.9Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hb59d6631923a902b
  [DEL] -30.0Ki  [DEL] -29.9Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hf69bf5e5176a850d
  [DEL] -30.6Ki  [DEL] -30.4Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::hc378838f03ecebc4
  [DEL] -30.6Ki  [DEL] -30.4Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h5c1589ffecbc9667
  [DEL] -30.7Ki  [DEL] -30.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::haefdb0cfbc155295
  [DEL] -38.3Ki  [DEL] -38.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h562fea205415e8e6
  [DEL] -41.4Ki  [DEL] -41.2Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h244a74d1ea5c869f
  [DEL] -56.7Ki  [DEL] -56.5Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h75350cf7be52522b
  +1.9%  +789Ki  +1.7%  +588Ki    TOTAL

@pr-commenter

pr-commenter Bot commented Jul 4, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 48d50611-4932-4d6f-b238-9ea5b11c265c
Baseline: 2715682b · Comparison: 0a7a0e62 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (3)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_low memory ⚪ +1.82 metrics profiles logs
quality_gates_rss_idle memory ⚪ +1.68 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +1.39 metrics profiles logs
Bounds Checks: ✅ Passed (3)
experiment check replicates observed links
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 43.8 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 65.5 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 29.3 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

webern added a commit that referenced this pull request Jul 4, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
@webern webern changed the base branch from m/pr3-dynamic to graphite-base/1979 July 4, 2026 16:08
@webern webern changed the base branch from graphite-base/1979 to m/pr3b-env-keys July 4, 2026 16:08

webern commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

@webern webern force-pushed the m/pr3b-env-keys branch from e6e28df to e6bfdc9 Compare July 4, 2026 16:10
@webern webern force-pushed the m/pr4-wired branch 2 times, most recently from 2523534 to 9353c90 Compare July 5, 2026 07:51
webern added a commit that referenced this pull request Jul 5, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
webern added a commit that referenced this pull request Jul 5, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
@webern webern force-pushed the m/pr3b-env-keys branch from e6bfdc9 to 74ecadf Compare July 5, 2026 11:55
webern added a commit that referenced this pull request Jul 5, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
webern added a commit that referenced this pull request Jul 5, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
webern added a commit that referenced this pull request Jul 5, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
webern added a commit that referenced this pull request Jul 5, 2026
## Human Summary

There's some weird stuff going on here with preserving "absence" in an
attempt to preserve exact behavioral parity, but I think it's OK.

## AI Summary

Cuts the OTLP components over from the raw `GenericConfiguration` map to
the typed
`SalukiConfiguration` model, following the PR #1979 pattern:
`OtlpConfig` (shared receiver
config), the OTLP source, decoder, forwarder, and relay.

Each component now builds from typed model slices (`domains.otlp`,
`domains.traces.otlp`)
instead of deserializing the raw config:

- **Source / relay / decoder / forwarder**: `from_configuration` takes
borrowed model slices;
the call site in `run.rs` reads them from the config system. Component
structs carry no serde
  and no defaults.
- **Seeded OTLP knobs** (context sizing, HTTP receiver transport, trace
interner size,
top-level-by-span-kind) are converted to the required-with-default
doctrine: non-optional in
`SalukiOnly` with the default defined once as a shared `const`/`fn` in
`agent-data-plane-config`
and referenced by both the model `Default` and the `SalukiOnly` serde
default. The two byte-size
interner keys are typed `ByteSize` so they accept both a bare integer
and a suffixed string.
- **`OtlpTracesTranslator`** now takes the two native booleans it
actually reads instead of the
whole raw `TracesConfig`. `TracesConfig` is kept only for the
not-yet-migrated Datadog trace
  encoder.
- The per-component config smoke tests are retired (coverage now lives
in the translator and
  construction tests).

### Behavior notes

Because the OTLP receiver keys are witnessed against the vendored
Datadog schema, their defaults
now come from that schema rather than Saluki's former hardcoded values.
When a key is left unset:

- gRPC/HTTP receiver endpoints default to `localhost:4317` /
`localhost:4318` (previously
  `0.0.0.0:...`), matching the Datadog Agent.
- `otlp_config.logs.enabled` defaults to `false` (previously `true`),
matching the Datadog Agent.
- `otlp_config.receiver.protocols.grpc.max_recv_msg_size_mib` of `0` now
maps to grpc-go's built-in
4 MiB limit, matching the Agent and the schema documentation (previously
`0` produced a 0-byte
  limit).

Deployments that set these keys explicitly (including the OTLP
correctness cases, which pin the
gRPC endpoint to `0.0.0.0:4317`) are unaffected.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?

- `make build-schema-overlay && make fmt` (no generated drift), `make
check-all`, `make check-docs`.
- Unit tests for `agent-data-plane-config`,
`agent-data-plane-config-system`, `saluki-components`,
and `agent-data-plane`, including new `SalukiOnly` transport/default
tests (both byte-size input
  forms) and per-component construction tests.
- Statically reviewed the OTLP correctness cases: all send over gRPC on
the explicitly configured
`0.0.0.0:4317`, so the receiver default changes above do not affect
them.

## References

- Progresses #1788
- Merges into `m/pr5-cutover`
@webern webern force-pushed the m/pr3b-env-keys branch from 67baf4b to 3a5cfe3 Compare July 8, 2026 15:21
webern added a commit that referenced this pull request Jul 8, 2026
## AI Summary

Build the OTTL filter processor from the resolved traces-domain
configuration instead of parsing `GenericConfiguration` in the
component. Remove the component-owned source serde configuration and use
the shared `OttlErrorMode` model type.

## Change Type
- [x] Non-functional (chore, refactoring, docs)

## How did you test this PR?
- `make build-schema-overlay && make fmt`
- `make check-all`
- `make test`
- `make check-docs`
- Targeted OTTL filter processor tests

## References

Related to configuration cutover PR #1979.
webern added a commit that referenced this pull request Jul 8, 2026
There's some weird stuff going on here with preserving "absence" in an
attempt to preserve exact behavioral parity, but I think it's OK.

Cuts the OTLP components over from the raw `GenericConfiguration` map to
the typed
`SalukiConfiguration` model, following the PR #1979 pattern:
`OtlpConfig` (shared receiver
config), the OTLP source, decoder, forwarder, and relay.

Each component now builds from typed model slices (`domains.otlp`,
`domains.traces.otlp`)
instead of deserializing the raw config:

- **Source / relay / decoder / forwarder**: `from_configuration` takes
borrowed model slices;
the call site in `run.rs` reads them from the config system. Component
structs carry no serde
  and no defaults.
- **Seeded OTLP knobs** (context sizing, HTTP receiver transport, trace
interner size,
top-level-by-span-kind) are converted to the required-with-default
doctrine: non-optional in
`SalukiOnly` with the default defined once as a shared `const`/`fn` in
`agent-data-plane-config`
and referenced by both the model `Default` and the `SalukiOnly` serde
default. The two byte-size
interner keys are typed `ByteSize` so they accept both a bare integer
and a suffixed string.
- **`OtlpTracesTranslator`** now takes the two native booleans it
actually reads instead of the
whole raw `TracesConfig`. `TracesConfig` is kept only for the
not-yet-migrated Datadog trace
  encoder.
- The per-component config smoke tests are retired (coverage now lives
in the translator and
  construction tests).

Because the OTLP receiver keys are witnessed against the vendored
Datadog schema, their defaults
now come from that schema rather than Saluki's former hardcoded values.
When a key is left unset:

- gRPC/HTTP receiver endpoints default to `localhost:4317` /
`localhost:4318` (previously
  `0.0.0.0:...`), matching the Datadog Agent.
- `otlp_config.logs.enabled` defaults to `false` (previously `true`),
matching the Datadog Agent.
- `otlp_config.receiver.protocols.grpc.max_recv_msg_size_mib` of `0` now
maps to grpc-go's built-in
4 MiB limit, matching the Agent and the schema documentation (previously
`0` produced a 0-byte
  limit).

Deployments that set these keys explicitly (including the OTLP
correctness cases, which pin the
gRPC endpoint to `0.0.0.0:4317`) are unaffected.

- [x] Non-functional (chore, refactoring, docs)

- `make build-schema-overlay && make fmt` (no generated drift), `make
check-all`, `make check-docs`.
- Unit tests for `agent-data-plane-config`,
`agent-data-plane-config-system`, `saluki-components`,
and `agent-data-plane`, including new `SalukiOnly` transport/default
tests (both byte-size input
  forms) and per-component construction tests.
- Statically reviewed the OTLP correctness cases: all send over gRPC on
the explicitly configured
`0.0.0.0:4317`, so the receiver default changes above do not affect
them.

- Progresses #1788
- Merges into `m/pr5-cutover`
@webern webern changed the base branch from m/pr3b-env-keys to graphite-base/1979 July 8, 2026 18:01
@webern webern force-pushed the graphite-base/1979 branch from 3a5cfe3 to 24621f3 Compare July 9, 2026 09:44
@webern webern changed the base branch from graphite-base/1979 to m/pr3b-env-keys July 9, 2026 09:44
@webern webern force-pushed the m/pr3b-env-keys branch from 24621f3 to cd6d290 Compare July 9, 2026 12:02
@webern webern force-pushed the m/pr3b-env-keys branch from cd6d290 to ed8a6cb Compare July 9, 2026 13:31
@webern webern force-pushed the m/pr3b-env-keys branch from ed8a6cb to 8ca4292 Compare July 9, 2026 14:27
@webern webern force-pushed the m/pr3b-env-keys branch from 8ca4292 to 90da4b6 Compare July 9, 2026 15:41
webern added 2 commits July 10, 2026 12:03
Cut the aggregate transform over from raw GenericConfiguration access to the
typed SalukiConfiguration model. AggregateConfiguration builds from the
dogstatsd aggregation slice and the shared histogram encoding; histogram
parsing moves into HistogramConfiguration::from_encoding, and the component
config struct no longer carries a Default or deserializes Datadog config.

The aggregation slice is made authoritative from the Agent schema, dropping
the redundant Saluki-only aggregation keys. run.rs reads the typed config and
hands the transform its subtree.
…onfig

Cut the debug-log destination over to the typed SalukiConfiguration model.
DogStatsDDebugLogConfiguration builds from the dogstatsd debug-log slice and
holds a Live view, so the destination reacts to metrics-stats changes through
the live view instead of a string-key watch. run.rs hands it the typed subtree
and mints the live view from the configuration system.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. area/docs Reference documentation. transform/aggregate Aggregate transform.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant