Skip to content

chore(core): split resource-accounting crate into resource tracking and bounds accounting - #2050

Merged
tobz merged 2 commits into
mainfrom
tobz/split-resource-accounting
Jul 9, 2026
Merged

chore(core): split resource-accounting crate into resource tracking and bounds accounting#2050
tobz merged 2 commits into
mainfrom
tobz/split-resource-accounting

Conversation

@tobz

@tobz tobz commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

This PR splits the resource-accounting crate into its two primary parts -- resource tracking and bounds accounting -- and moves them to saluki-common and saluki-core, respectively.

As part of the refactoring work around SubsystemIdentifier (#2029, #2048), we want to utilize SubsystemIdentifier for all things that generate structured identifiers that are attached to "components", like topology components or the environment provider and so on. Part of this is utilizing SubsystemIdentifier instead of utilizing opaque string types so that we can enforce invariants -- only certain characters in identifiers, etc -- at the type system. However, to do this for the resource accounting component registry (resource_accounting::ComponentRegistry) means depending on saluki-core where SubsystemIdentifier is defined... which ends up as a circular dependency. Doh!

In preparation of moving SubsystemIdentifier to saluki-common, we need to split out the parts of resource-accounting that saluki-common depends on, and as it so happens, there's already a natural "seam", as Claude would call it, between the tracking of resource usage (allocations, CPU) and the accounting of expected usage (memory bounds, bounds verification, etc).

This PR moves all resource tracking-related bits into saluki-common to satisfy not only its own usages of those types, but also acts as the non-circular dependency on which other crates can depend to utilize those types as well. Since memory bounds-related bits are (at least as of right now) only utilized in higher-level crates where components and such are defined, we've moved those to saluki-core as all crates that have types with defined bounds already import saluki-core. This resolves all of our dependency tensions and unlocks the ability for us to move ComponentRegistry to utilize SubsystemIdentifier.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

Existing tests.

References

DADP-2

@tobz tobz added the type/chore Updates to dependencies or general "administrative" tasks necessary to maintain the codebase/repo. label Jul 9, 2026
@tobz
tobz requested a review from a team as a code owner July 9, 2026 13:22
@dd-octo-sts dd-octo-sts Bot added area/core Core functionality, event model, etc. area/config Configuration. area/components Sources, transforms, and destinations. area/memory Memory bounds and memory management. area/ci CI/CD, automated testing, etc. source/dogstatsd DogStatsD source. transform/aggregate Aggregate transform. transform/chained Chained transform. transform/host-enrichment Host Enrichment synchronous transform. destination/blackhole Blackhole destination. destination/prometheus Prometheus Scrape destination. area/observability Internal observability of ADP and Saluki. destination/dogstatsd-stats DogStatsD Statistics destination. transform/dogstatsd-mapper DogStatsD Mapper synchronous transform. source/heartbeat Heartbeat source. encoder/buffered-incremental Buffered Incremental encoder. encoder/datadog-events Datadog events encoder. encoder/datadog-logs Datadog Logs encoder. encoder/datadog-metrics Datadog Metrics encoder. encoder/datadog-service-checks Datadog Service Checks encoder. encoder/datadog-stats Datadog APM Stats encoder. encoder/datadog-traces Datadog Traces encoder. forwarder/datadog Datadog forwarder. source/otlp OTLP source. transform/apm-stats APM Stats transform. transform/metric-router Metric Router transform. labels Jul 9, 2026
@dd-octo-sts dd-octo-sts Bot added transform/trace-obfuscation Trace Obfuscation synchronous transform. transform/trace-sampler Trace Sampler synchronous transform. decoders/otlp forwarder/otlp relay/otlp labels Jul 9, 2026
@pr-commenter

pr-commenter Bot commented Jul 9, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: dbece44 · Comparison: dbb35c8 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.66 MiB (baseline) vs 41.53 MiB (comparison)
Size Change: -136.80 KiB (-0.32%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
resource_accounting -121.13 KiB 132
saluki_core::accounting::registry +58.66 KiB 70
core -48.64 KiB 15954
saluki_common::resource_tracking::groups +26.13 KiB 26
alloc -17.35 KiB 3107
matchit -14.65 KiB 56
anon.9d74b0aa49281f31c2e13ca4fdb52337.65.llvm.9026379143731643429 -10.33 KiB 1
hashbrown -10.29 KiB 1632
anon.20d40124d2abbd7d161bca533ae2bc4d.74.llvm.3881346718721636546 +10.24 KiB 1
[sections] -8.71 KiB 7
serde_json +8.58 KiB 360
anon.af29a6a9d4a9362117e0fbd5d9b09a76.66.llvm.13283325591066943712 +8.08 KiB 1
anon.c4832a9e941eede0b3fac8601fc5dbed.149.llvm.16028478477781504075 -7.98 KiB 1
saluki_core::accounting::limiter +7.62 KiB 12
saluki_core::accounting::ComponentBounds +7.33 KiB 3
anon.7795f5375d38aceee11397747b095f1d.30.llvm.2564832767356818503 -7.24 KiB 1
anon.0a98b3590f538e02e15153f12102fa90.2.llvm.17224994202189465113 +7.07 KiB 1
tokio -6.75 KiB 5277
saluki_common::cache::Cache<K,V,W,H> -6.40 KiB 26
saluki_common::task::instrument -4.96 KiB 58
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +56.7Ki  [NEW] +56.5Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h75350cf7be52522b
  [NEW] +47.9Ki  [NEW] +47.8Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h244a74d1ea5c869f
  [NEW] +38.3Ki  [NEW] +38.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h562fea205415e8e6
  [NEW] +32.7Ki  [NEW] +32.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h0399488f11e43c11
  [NEW] +30.7Ki  [NEW] +30.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::haefdb0cfbc155295
  [NEW] +30.6Ki  [NEW] +30.4Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h5c1589ffecbc9667
  [NEW] +30.6Ki  [NEW] +30.4Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::hc378838f03ecebc4
  [NEW] +30.3Ki  [NEW] +30.2Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hf69bf5e5176a850d
  [NEW] +28.0Ki  [NEW] +27.9Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hb59d6631923a902b
  [NEW] +27.4Ki  [NEW] +27.1Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hf7c926a0582af9a6
  [DEL] -27.4Ki  [DEL] -27.1Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hdaa8cb1a5a827368
  [DEL] -28.0Ki  [DEL] -27.9Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h91635d5deaa854cc
  [DEL] -30.2Ki  [DEL] -30.1Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h59e759ccdf259361
  [DEL] -30.6Ki  [DEL] -30.4Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h8a1b0fac51f48708
  [DEL] -30.6Ki  [DEL] -30.4Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h748958f492a43cff
  [DEL] -30.7Ki  [DEL] -30.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h7d6dc24e0d68ba5f
  [DEL] -32.7Ki  [DEL] -32.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h12f67e29ed7b9186
  [DEL] -38.5Ki  [DEL] -38.3Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::haae88f8b6cc919ec
  [DEL] -47.8Ki  [DEL] -47.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hf0f541f4d97eee22
  [DEL] -56.7Ki  [DEL] -56.5Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h14ea2d62dcf96940
  -0.6%  -136Ki  -0.7%  -114Ki    [56010 Others]
  -0.3%  -136Ki  -0.3%  -114Ki    TOTAL

@datadog-datadog-prod-us1-2

This comment has been minimized.

@pr-commenter

pr-commenter Bot commented Jul 9, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 25cf6db1-7aa5-4d2d-acb7-28525ce3e4c8
Baseline: dbece44f · Comparison: dbb35c85 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_ultraheavy memory ⚪ +0.05 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.14 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ -0.18 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ -0.24 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.32 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 138 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 43 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 64.8 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 192 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 29 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@lucastemb lucastemb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@tobz
tobz merged commit bfc26f6 into main Jul 9, 2026
83 checks passed
@tobz
tobz deleted the tobz/split-resource-accounting branch July 9, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci CI/CD, automated testing, etc. area/components Sources, transforms, and destinations. area/config Configuration. area/core Core functionality, event model, etc. area/memory Memory bounds and memory management. area/observability Internal observability of ADP and Saluki. decoders/otlp destination/blackhole Blackhole destination. destination/dogstatsd-stats DogStatsD Statistics destination. destination/prometheus Prometheus Scrape destination. encoder/buffered-incremental Buffered Incremental encoder. encoder/datadog-events Datadog events encoder. encoder/datadog-logs Datadog Logs encoder. encoder/datadog-metrics Datadog Metrics encoder. encoder/datadog-service-checks Datadog Service Checks encoder. encoder/datadog-stats Datadog APM Stats encoder. encoder/datadog-traces Datadog Traces encoder. forwarder/datadog Datadog forwarder. forwarder/otlp relay/otlp source/dogstatsd DogStatsD source. source/heartbeat Heartbeat source. source/otlp OTLP source. transform/aggregate Aggregate transform. transform/apm-stats APM Stats transform. transform/chained Chained transform. transform/dogstatsd-mapper DogStatsD Mapper synchronous transform. transform/host-enrichment Host Enrichment synchronous transform. transform/metric-router Metric Router transform. transform/trace-obfuscation Trace Obfuscation synchronous transform. transform/trace-sampler Trace Sampler synchronous transform. type/chore Updates to dependencies or general "administrative" tasks necessary to maintain the codebase/repo.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants