fix(dogstatsd): decouple datagram reads from decoding and increase buffer limit to 256MiB

[aqian01]

Summary

Issue
Improve ADP's UDP and UDS datagram burst handling. Socket reads were coupled to decoding, and ADP's default elastic buffer ceiling was about 2 MiB and raised to match the Core Agent's default of roughly 256 MiB.

This PR:

• Queues connectionless socket reads in a bounded global buffer pool and decodes them in parallel.
• Uses a one-item acknowledged channel for connected streams, preserving order without building a backlog.
• Supports dogstatsd_workers_count. When unset or zero, ADP uses max(vCPUs - 2, 2), matching the Core Agent's default formula. This count configures one global decoder worker pool shared by all connectionless UDP and UDS listeners. Connected streams continue to use one decoder per connection.
• Raises the elastic ceiling from 256 to 32,768 buffers of about 8 KiB each—approximately 2 MiB to 256 MiB—while keeping the initial allocation at 128 buffers.
• Shares the elastic buffer pool across connectionless listeners and connected readers. Connectionless datagrams wait in the bounded global queue, while connected streams use one-item acknowledged channels.
• Reclaims excess idle buffers after bursts while retaining a small warm pool.

Benchmark

Original benchmark: baseline ADP

The original DADP-150 benchmark used the pre-PR ADP behavior: sequential decoding with the original ~2 MiB buffer ceiling. It ran for 180 seconds with 60 clients, estimated 32 MiB bursts, client aggregation, and background sending. It reported bytes_dropped_writer / (bytes_dropped_writer + bytes_sent).

Burst interval	Core datagram	Core stream	ADP datagram	ADP stream
15s	3.9%	0.5%	13.9%	3.9%
30s	5.1%	0.5%	9.7%	2.2%
60s	4.4%	0.6%	17.9%	6.2%

Original benchmark: decoupled reads from decoding

The original benchmark was rerun on four CPUs against the Core Agent, baseline ADP, and an early PR build that decoupled socket reads from decoding but retained one decoder and the original ~2 MiB buffer ceiling. Datagram results were:

Burst interval	Core Agent	Baseline ADP	Decoupled reads from decoding (~2 MiB)
15s	11.4%	59.8%	52.7%
30s	9.2%	51.5%	48.5%
60s	14.2%	52.8%	51.6%
Unweighted mean	11.6%	54.7%	50.9%

So decoupling socket reads from decoding was an incremental change but did not close the gap between ADP and the agent

Original benchmark: parallel decoding with larger buffers

The original benchmark was rerun on four CPUs against the Core Agent, baseline ADP, and a PR build that combined parallel decoding with the increased ~256 MiB buffer ceiling. Datagram results were:

Burst interval	Core Agent	Baseline ADP	Parallel decoding + ~256 MiB
15s	41.8%	82.5%	6.8%
30s	20.9%	48.5%	4.1%
60s	28.3%	48.0%	7.7%
Unweighted mean	30.3%	59.7%	6.2%

The larger buffer produced a substantial improvement in this run. However, five of the nine final counter snapshots were mathematically inconsistent, and the 15s PR result required one retry after the first run produced no telemetry. These results are directional evidence only.

Why the benchmark changed

The benchmark was changed because:

• It was time-based, so contenders could complete different amounts of work. In the 30s case, the original logs captured 45,982,301 attempted bytes for ADP versus 36,695,095 for Core—25.3% more traffic during the same 180-second window. Attempted bytes are calculated as bytes_sent + bytes_dropped_writer.
• The original collector and logs queried counters sequentially while traffic continued, producing non-atomic snapshots. One sample reported bytes_dropped=2,541,636 below bytes_dropped_writer=2,762,874, which violates the counter definition.
• The original collector forcibly terminated workers without a drain step or verifying that their background queues were empty.
• Each case ran once, so variance was unknown.

Fixed workload: parallel decoding with larger buffers

The revised benchmark tested parallel decoding with the buffer ceiling increased from ~2 MiB to ~256 MiB. It used a fixed workload instead of a fixed duration. Each contender received the same load: 60 clients sending three 32 MiB bursts over UDS datagrams, with four CPUs, client aggregation, and background sending. After the workload completed, client queues were drained before lifetime counters were collected.

The reported percentage is the share of attempted packet bytes that clients failed to send. These failures are consistent with receiver backpressure, although the harness does not capture the specific socket errors.

Burst interval	Core Agent	Parallel decoding + ~256 MiB	Core elapsed	ADP elapsed
15s	18.1%	8.7%	43.7s	42.6s
30s	8.6%	2.5%	70.7s	70.6s
60s	8.7%	2.6%	130.5s	130.2s
Unweighted mean	11.8%	4.6%	—	—

With the fixed-workload harness, parallel decoding with a ~32 MiB ceiling averaged 39.8%. Increasing the ceiling to ~256 MiB reduced the average to 4.6%. Each case was run once, so variance remains unknown.

These fixed-workload results were produced at commit b0a95a65, before the idle-buffer reclamation change. They are directional evidence, not direct validation of the current head.

Download the benchmark scripts, configuration, pinned dependencies, logs, and JSON summaries.

Memory trade-off

Decoupling socket reads from decoding lets ADP continue draining the socket instead of making senders wait for capacity. This moves more of the backlog into ADP's userspace buffer pool, trading higher memory usage for less backpressure and fewer packet drops during bursts.

The pool allocates 128 buffers initially and can grow on demand to 32,768 buffers, about 256 MiB of payload backing capacity. After five seconds without growth, it shrinks toward 256 idle buffers, so the full ceiling is not normally retained. The performance report shows the expected trade-off: medium-load RSS increased 22.31% to 83.3 MiB, above its 75 MiB bound, while low and idle loads remained within bounds.

To test whether reduced backpressure explains the higher RSS, three isolated three-minute quality_gates_rss_dsd_medium runs measured Lading's cumulative send().await duration. Lading waits for the UDS socket to accept the datagram, while the original benchmark client used 10 ms socket and queue timeouts that could turn backpressure into counted drops. Longer send duration therefore indicates more sender backpressure. Each run sent about 487,000 packets.

Case	Buffer ceiling	Send duration	Average RSS	Average pool size
Baseline ADP (sequential decoding)	~2 MiB	5.91s	58.9 MiB	~1.0 MiB
Parallel decoding only	~2 MiB	5.34s	61.3 MiB	~2.0 MiB
Parallel decoding + larger buffer	~256 MiB	4.78s	67.0 MiB	~5.1 MiB

Compared with baseline ADP, parallel decoding with the ~256 MiB ceiling spent 1.13 seconds less in send().await and used 8.0 MiB more RSS. Running the same parallel-decoding build with the original ~2 MiB ceiling increased send duration by 0.57 seconds and reduced RSS by 5.7 MiB. This supports the explanation that reduced socket backpressure shifts backlog into ADP memory. Parallel decoding alone remained 2.3 MiB above baseline, so other factors also contribute. Each case was run once, so variance is unknown.

Correctness test harness

Decoupled datagram processing can leave packets queued for separate decoder workers. Origin enrichment uses the sender PID from UDS credentials, but the harness previously exited Millstone before flushing and collecting. It now reports completion while keeping Millstone alive until collection finishes. This prevents the sender-PID lifecycle race without changing the workload or assertions.

Validation

• make fmt
• 139 DogStatsD tests passed
• cargo check --workspace
• cargo check --workspace --tests
• make check-all

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 42b3b21 · Comparison: 94cad96 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.29 MiB (baseline) vs 41.60 MiB (comparison)
Size Change: +316.54 KiB (+0.75%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
tracing	+156.41 KiB	37
saluki_common::task::instrument	-106.10 KiB	35
figment	+91.63 KiB	74
otlp_protos::otlp_include::opentelemetry	+70.49 KiB	100
tonic_prost	-58.28 KiB	15
tokio	+57.99 KiB	597
saluki_components::sources::dogstatsd	+37.15 KiB	207
core	+31.11 KiB	2462
hyper_util	+30.42 KiB	18
serde	+24.62 KiB	31
[sections]	+20.23 KiB	8
anon.42d46170f28cdd82ce03344dec33bac2.801.llvm.12992742372782925659	+17.80 KiB	1
anon.55855941dea09b8f86cdb13230ac3b0e.801.llvm.14993132552406233135	-17.80 KiB	1
anon.5ffef35c7768023811ac5ab5fd4f00ff.888.llvm.7247732555510565809	+17.53 KiB	1
anon.5ffef35c7768023811ac5ab5fd4f00ff.885.llvm.15862285824570942458	-17.45 KiB	1
hyper	-17.42 KiB	64
http_body_util	+14.30 KiB	43
anon.612e9d0af0dace76c4c28517e517b344.706.llvm.9963701042273717628	+12.28 KiB	1
anon.509106849e6d222789cb4be5fe084148.496.llvm.1681420877705664797	-12.28 KiB	1
&mut serde_json	-11.26 KiB	30

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +2.4%  +249Ki  +2.2%  +177Ki    [16516 Others]
  [NEW] +21.4Ki  [NEW] +21.2Ki    saluki_components::sources::dogstatsd::drive_decoder::_{{closure}}::h26a33e23ae032832
  [NEW] +21.1Ki  [NEW] +21.0Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::hbef42a8b1492df11
  [NEW] +20.4Ki  [NEW] +20.3Ki    saluki_components::transforms::trace_obfuscation::TraceObfuscation::obfuscate_span::hdb90b509c50ba377
  [NEW] +18.9Ki  [NEW] +18.7Ki    _<hyper_util::server::conn::auto::Connection<I,S,E> as core::future::future::Future>::poll::h40c9bbc3582a5d19
  [NEW] +18.6Ki  [NEW] +18.4Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::h4481e0171fc4f085
  [NEW] +18.0Ki  [NEW] +17.8Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::hf77307ff9a3375eb
  [NEW] +17.9Ki  [NEW] +17.8Ki    h2::proto::connection::Connection<T,P,B>::poll::hf55737dd7d0240e8
  [NEW] +17.8Ki  [NEW]     +76    anon.42d46170f28cdd82ce03344dec33bac2.801.llvm.12992742372782925659
  [NEW] +17.6Ki  [NEW] +17.4Ki    saluki_components::transforms::apm_stats::span_concentrator::SpanConcentrator::flush::ha184bfce3e1498d8
  [NEW] +17.5Ki  [NEW] +17.4Ki    anon.5ffef35c7768023811ac5ab5fd4f00ff.888.llvm.7247732555510565809
  [NEW] +17.2Ki  [NEW] +17.1Ki    h2::proto::connection::Connection<T,P,B>::poll::hde2b4edde49019aa
  [NEW] +17.1Ki  [NEW] +17.0Ki    h2::proto::connection::Connection<T,P,B>::poll::h8dffaa99d2c44419
  [DEL] -15.9Ki  [DEL] -15.4Ki    core::ptr::drop_in_place<datadog_agent_config::generated::datadog_configuration::DatadogConfiguration>::h1fc411b0a78628dd
  [DEL] -17.4Ki  [DEL] -17.4Ki    anon.5ffef35c7768023811ac5ab5fd4f00ff.885.llvm.15862285824570942458
  [DEL] -17.8Ki  [DEL]     -76    anon.55855941dea09b8f86cdb13230ac3b0e.801.llvm.14993132552406233135
  [DEL] -18.3Ki  [DEL] -18.0Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::hfdfd1ef0ed851a0a
  [DEL] -18.9Ki  [DEL] -18.7Ki    _<tonic_prost::codec::ProstEncoder<T> as tonic::codec::Encoder>::encode::he25e149f9ca0e5a5
  [DEL] -19.6Ki  [DEL] -19.5Ki    saluki_components::transforms::apm_stats::ApmStats::process_trace::hce992ca7b6928d1d
  [DEL] -21.2Ki  [DEL] -21.0Ki    _<saluki_components::sources::dogstatsd::DogStatsDConfiguration as saluki_core::components::sources::builder::SourceBuilder>::build::_{{closure}}::hea5210e3bf2131b7
  [DEL] -27.2Ki  [DEL] -27.1Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::hb18fa47632f24940
  +0.7%  +316Ki  +0.7%  +244Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 32ab9ea2-a4eb-414c-928d-24f18c52d74a
Baseline: 42b3b216 · Comparison: 94cad961 · diff

Optimization Goals: ❌ 3 regressions detected

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	🔴 +64.76	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	🔴 +48.33	metrics profiles logs
quality_gates_rss_dsd_medium	memory	🔴 +22.37	metrics profiles logs

Fine details of change detection per experiment (2)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +2.99	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.36	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 227 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 51.1 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 89.2 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 391 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 32 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[webern]

This stood up to scrutiny...

Some interesting findings:

 Potential future performance work, only if profiling justifies it:
 - recvmmsg to receive multiple datagrams per syscall,

Why run the new reader/channel/oneshot architecture for connection-oriented streams when it provides no pipelining? Is the added unbounded per-connection task/channel overhead acceptable, or should connection-oriented streams retain a specialized inline path?

[aqian01]

dadp150-pr2079-benchmark-bundle.zip

Modified benchmark script

[datadog-datadog-prod-us1]

Connectionless readers no longer retain their promised reserved buffer while idle. A set of connection-oriented clients that leave partial frames open can consume the shared pool ceiling, preventing UDP/UDS datagram reads and causing the packet loss this change is intended to reduce.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit fa4e191 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-datadog-prod-us1]

Reserve buffers for connectionless readers

Mixed UDP/UDS-datagram and stream deployments can lose all connectionless DogStatsD traffic under partial-frame stream pressure.

Assertion details

• Input: A deployment enables a UDP or Unix-datagram listener alongside a TCP or Unix-stream listener and has its pool ceiling occupied by connection-oriented clients that each send an unterminated partial frame.
• Expected: Each connectionless listener retains at least one pool slot, so it can continue draining its socket regardless of stalled connected streams.
• Actual: After an idle datagram read completes, its buffer is returned to the shared pool. Connected readers can hold every buffer while awaiting more bytes, so the next datagram read blocks at take_buffer and the kernel socket queue eventually drops packets.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[aqian01]

With the default 32,768 buffers, this would require roughly 32,000 stalled connections so it's unlikely enough that I don’t consider it a blocker.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fa4e1919d5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Resolve UDS datagram origins before queueing

When UDS datagram origin detection or traffic capture is enabled, this queues only the payload plus peer_addr credentials and defers capture_uds_traffic/handle_frame until a decoder worker runs. Under bursts, a short-lived sender can exit while its packet is waiting in this channel; the later live-PID lookup can then lose or misattribute the container/origin tags even though the kernel supplied the PID at receive time. Resolve/cache the origin or capture container ID in the reader before enqueueing so queued datagrams do not depend on the sender still being alive.

Useful? React with 👍 / 👎.

[aqian01]

Resolved. The socket reader now resolves and pins the process origin before enqueueing, and decoder/capture paths use that pinned entity. Added a PID-reuse regression test.

[pr-commenter]

DogStatsD RSS Comparison (Core Agent vs ADP)

Run ID: 143469bd-9588-4805-9eb9-0d0bc6908d89
Baseline: Core Agent 7.81.3-full · Comparison: ADP de14266a6be56be90be5617407546aaa09be7338

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dogstatsd_rss_core_vs_adp_ultraheavy	memory	🟢 -62.14	metrics profiles logs
dogstatsd_rss_core_vs_adp_heavy	memory	🟢 -64.06	metrics profiles logs
dogstatsd_rss_core_vs_adp_medium	memory	🟢 -70.42	metrics profiles logs
dogstatsd_rss_core_vs_adp_low	memory	🟢 -74.23	metrics profiles logs
dogstatsd_rss_core_vs_adp_idle	memory	🟢 -75.15	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.