feat(metrics): support v3 payload splitting#1758
Conversation
rayz
changed the title
dd-octo-sts
Bot
added
area/components
This comment has been minimized.
This comment has been minimized.
rayz
force-pushed
the
rayz/metrics-v3-payload-splitting-clean
branch
from
1a4fe2d to
4989436
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4989436a02
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let v3_payload_limits = V3PayloadLimits::new( | ||
| self.max_series_payload_size, | ||
| self.max_series_uncompressed_payload_size, |
There was a problem hiding this comment.
When V3 sketches are enabled, this single v3_payload_limits value is also used by encode_and_flush_v3_sketch_metrics, so sketch splitting/dropping is controlled by serializer_max_series_payload_size and serializer_max_series_uncompressed_payload_size rather than the generic sketch limits. In a config that only intends to constrain series payloads (for example setting the series max to 0 or lower than the generic limit), V3 sketch payloads are unexpectedly split or dropped even though serializer_max_payload_size would allow them.
Useful? React with 👍 / 👎.
| if metric_points == 0 { | ||
| // The Agent drops zero-point V3 metrics before writing them. | ||
| context.telemetry.events_dropped_encoder().increment(1); | ||
| continue; |
There was a problem hiding this comment.
For zero-point metrics that occur after a non-empty metric, this continue leaves current_start unchanged, so the later range (for example 0..metrics.len()) still includes the metric that was just counted as dropped. That means a batch like [valid_metric, empty_metric] still encodes and reports the empty V3 metric instead of dropping it, while also incrementing the dropped counter.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds V3 metrics payload splitting so a single flush can produce multiple HTTP requests while enforcing payload size and point-count limits.
Changes:
- Introduces V3 payload limit/request helper types.
- Splits V3 series and sketch flushes by point count and encoded payload size.
- Adds tests for V3 splitting and batch header propagation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
lib/saluki-components/src/encoders/datadog/metrics/v3/payload.rs |
Adds V3 payload limit and request metadata structs. |
lib/saluki-components/src/encoders/datadog/metrics/v3/mod.rs |
Exposes the new V3 payload module internally. |
lib/saluki-components/src/encoders/datadog/metrics/mod.rs |
Wires V3 payload limits into the encoder and implements split/flush behavior with tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| let v3_payload_limits = V3PayloadLimits::new( | ||
| self.max_series_payload_size, | ||
| self.max_series_uncompressed_payload_size, | ||
| self.max_metrics_per_payload, | ||
| SERIES_V3_POINTS_PER_PAYLOAD_LIMIT, | ||
| ); |
| // The Agent drops zero-point V3 metrics before writing them. | ||
| context.telemetry.events_dropped_encoder().increment(1); |
Binary Size Analysis (Agent Data Plane)Baseline: 1ad7cde · Comparison: 608d534 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (35)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
rayz
merged commit 56bd735
into
tobz/datadog-metrics-v3-payload-support
Summary
Adds V3 metrics payload splitting so ADP no longer assumes each V3 flush produces a single request.
The encoder now enforces V3 series/sketch payload limits, drops unsendable zero-point or oversized metrics, and emits correct
X-Metrics-Request-Seq/X-Metrics-Request-Lenheaders when a flush produces multiple requests.Change Type
How did you test this PR?
unit tests / ci
References