enhancement(antithesis): DogStatsD load generation#1782
Open
blt wants to merge 2 commits into
Open
Conversation
This commit introduces DogStatD load generation. If you're familiar with my work in lading you might notice this is very different. The implementation here is responsible for giving Antithesis choices between 'compliant' dogstatsd lines and 'feral' lines which are _mechanically_ allowable lines but are in practice not sensible. Also of interest control of contexts and send rates are jettisoned entirely, that's an antithesis scheduling concern. Metrics, events and service-checks are all present. Depending on the antithesis shots I may need to adjust the ratios of compliant to feral outcomes.
dd-octo-sts
Bot
added
the
area/test
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
4 tasks
|
Binary Size Analysis (Agent Data Plane)Baseline: 1bd1613 · Comparison: a31a487 · diff ✅ Binary size difference within thresholdChanges by Module
Detailed Symbol Changes |
Regression Detector (Agent Data Plane)Run ID: Optimization Goals: ✅ No significant changes detectedFine details of change detection per experiment (35)Experiments configured
Bounds Checks: ✅ Passed (5)
ExplanationA change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression ( |
Signed-off-by: Brian L. Troutwine <brian.troutwine@datadoghq.com>
blt
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a31a487f1b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| pub(crate) fn write_segments<R: Rng + ?Sized>( | ||
| rng: &mut R, buf: &mut Vec<u8>, vibe: Vibe, compliant: &[&[u8]], aberrant: &[&[u8]], separators: &[u8], | ||
| ) { | ||
| let count = Boundary::<u8>::new().sample(rng); |
There was a problem hiding this comment.
When vibe is Vibe::Clean, this boundary sampler can return 0, so write_words emits an empty metric name, service-check name, event title, or event text. Those packets are not well-formed in the codebase's parsers (take_while1 for metric/service names and an explicit zero-length reject for events), so a supposedly clean batch can still be dropped before reaching the intake; this makes the delivery baseline and the "fully clean batch" assertion much weaker than intended. Use a non-zero count for clean identifier/text fields and reserve zero-length fields for feral cases.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR adds an Antithesis-oriented DogStatsD workload generator to the test harness, shifting the Antithesis setup away from the prior millstone-based load driver and toward generating a mix of “clean” (compliant) and “feral” (mechanically allowable but nonsensical) DogStatsD lines (metrics/events/service-checks). It also updates the Antithesis scratchbook documentation to reflect the new workload and its current bug-reproduction reach.
Changes:
- Add a new
harness::payload::dogstatsdmodule that generates metric, event, and service-check lines with clean/feral “vibes”. - Rework
parallel_driver_send_dogstatsdto send randomly sized batches of generated DogStatsD messages and update associated SDK assertions. - Update Antithesis deployment and scratchbook docs for the new workload driver and topology.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| test/antithesis/scratchbook/property-catalog.md | Updates catalog notes/status to reflect new driver behavior and workload reach. |
| test/antithesis/scratchbook/existing-assertions.md | Updates assertion inventory and explains the new driver; needs minor wording alignment. |
| test/antithesis/scratchbook/deployment-topology.md | Updates topology narrative for the new driver; contains an outdated UDS note vs actual deployment. |
| test/antithesis/scratchbook/bug-ledger.md | Adds “workload reach” note and updates bug-hunt entry details. |
| test/antithesis/harness/src/rand.rs | Extends randomness helpers with Probe for multiple numeric types and Boundary<T> samplers. |
| test/antithesis/harness/src/payload/dogstatsd/common.rs | Adds shared “segment/number/tag” construction utilities and clean/feral vibe handling. |
| test/antithesis/harness/src/payload/dogstatsd/metrics.rs | Adds feral metric line generator with randomized extensions/tags/value encodings. |
| test/antithesis/harness/src/payload/dogstatsd/events.rs | Adds feral event generator (including skewed header lengths in feral mode). |
| test/antithesis/harness/src/payload/dogstatsd/service_checks.rs | Adds feral service-check generator. |
| test/antithesis/harness/src/payload/dogstatsd.rs | Adds top-level DogStatsD message selection and in-file format reference notes. |
| test/antithesis/harness/src/payload.rs | Adds payload module root. |
| test/antithesis/harness/src/lib.rs | Exposes the new payload module from the harness crate. |
| test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs | Replaces old regime-based metric sender with batch-based clean/feral/mixed driver. |
| test/antithesis/harness/src/bin/first_sample_config/config.rs | Changes sampled log_level values; currently introduces an unsupported level for ADP. |
| test/antithesis/harness/Cargo.toml | Adds itoa/ryu dependencies for fast numeric formatting. |
| test/antithesis/deploy/Dockerfile | Removes millstone from workload image build; builds/copies new test-command binaries. |
| test/antithesis/deploy/docker-compose.yaml | Removes RUST_BACKTRACE env var; configures shared UDS volume for DogStatsD socket. |
| Cargo.lock | Records new dependency edges for harness additions (itoa, ryu). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
66
to
+70
| pub(crate) enum LogLevel { | ||
| /// Warnings and above. | ||
| Warn, | ||
| /// Errors only. | ||
| Error, | ||
| /// Critical only — the quietest level that still logs. | ||
| Critical, |
| pub(crate) fn write_segments<R: Rng + ?Sized>( | ||
| rng: &mut R, buf: &mut Vec<u8>, vibe: Vibe, compliant: &[&[u8]], aberrant: &[&[u8]], separators: &[u8], | ||
| ) { | ||
| let count = Boundary::<u8>::new().sample(rng); |
Comment on lines
60
to
65
| @@ -65,12 +65,13 @@ Notes: | |||
| sidecar — see "Listener-coverage variant".) | |||
| | `test/antithesis/harness/src/bin/eventually_adp_alive.rs:62` | `assert_always!` | "ADP booted: API reachable and DogStatsD socket present" | harness binary (`eventually_`, faults-paused) | Death-liveness for `adp-stays-alive` — fails the branch when ADP self-crashed (config panic / load) but stayed down through the quiet period. | | ||
| | `lib/saluki-components/src/common/datadog/io.rs:553` | `assert_sometimes!` | "ADP forwarded a payload to the intake" (`{ domain }`) | `#[cfg(feature = "antithesis")]` | First in-SUT property assertion — good-function liveness (the full pipeline ran to a 2xx) + replay checkpoint; good-function half of `adp-keeps-delivering`, in-SUT seed of `forwarder-eventual-delivery`. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:46` | `assert_reachable!` | "workload ran a dogstatsd batch" | harness binary | Confirms the DSD driver ran a batch; details carry the attempted-line count and socket path. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:53` | `assert_sometimes!` | "workload delivered at least one dogstatsd line" (`attempted > 0`) | harness binary | Anti-vacuity anchor: a batch can sample count == 0, so "ran" does not imply "sent"; this proves a timeline sometimes actually delivers a line, else delivery checks are vacuous. | |
Comment on lines
66
to
+70
| pub(crate) enum LogLevel { | ||
| /// Warnings and above. | ||
| Warn, | ||
| /// Errors only. | ||
| Error, | ||
| /// Critical only — the quietest level that still logs. | ||
| Critical, |
| | `test/antithesis/harness/src/bin/eventually_adp_alive.rs:62` | `assert_always!` | "ADP booted: API reachable and DogStatsD socket present" | harness binary (`eventually_`, faults-paused) | Death-liveness for `adp-stays-alive` — fails the branch when ADP self-crashed (config panic / load) but stayed down through the quiet period. | | ||
| | `lib/saluki-components/src/common/datadog/io.rs:553` | `assert_sometimes!` | "ADP forwarded a payload to the intake" (`{ domain }`) | `#[cfg(feature = "antithesis")]` | First in-SUT property assertion — good-function liveness (the full pipeline ran to a 2xx) + replay checkpoint; good-function half of `adp-keeps-delivering`, in-SUT seed of `forwarder-eventual-delivery`. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:46` | `assert_reachable!` | "workload ran a dogstatsd batch" | harness binary | Confirms the DSD driver ran a batch; details carry the attempted-line count and socket path. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:53` | `assert_sometimes!` | "workload delivered at least one dogstatsd line" (`attempted > 0`) | harness binary | Anti-vacuity anchor: a batch can sample count == 0, so "ran" does not imply "sent"; this proves a timeline sometimes actually delivers a line, else delivery checks are vacuous. | |
Comment on lines
66
to
+70
| pub(crate) enum LogLevel { | ||
| /// Warnings and above. | ||
| Warn, | ||
| /// Errors only. | ||
| Error, | ||
| /// Critical only — the quietest level that still logs. | ||
| Critical, |
| pub(crate) fn write_segments<R: Rng + ?Sized>( | ||
| rng: &mut R, buf: &mut Vec<u8>, vibe: Vibe, compliant: &[&[u8]], aberrant: &[&[u8]], separators: &[u8], | ||
| ) { | ||
| let count = Boundary::<u8>::new().sample(rng); |
Comment on lines
60
to
65
| @@ -65,12 +65,13 @@ Notes: | |||
| sidecar — see "Listener-coverage variant".) | |||
| | `test/antithesis/harness/src/bin/eventually_adp_alive.rs:62` | `assert_always!` | "ADP booted: API reachable and DogStatsD socket present" | harness binary (`eventually_`, faults-paused) | Death-liveness for `adp-stays-alive` — fails the branch when ADP self-crashed (config panic / load) but stayed down through the quiet period. | | ||
| | `lib/saluki-components/src/common/datadog/io.rs:553` | `assert_sometimes!` | "ADP forwarded a payload to the intake" (`{ domain }`) | `#[cfg(feature = "antithesis")]` | First in-SUT property assertion — good-function liveness (the full pipeline ran to a 2xx) + replay checkpoint; good-function half of `adp-keeps-delivering`, in-SUT seed of `forwarder-eventual-delivery`. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:46` | `assert_reachable!` | "workload ran a dogstatsd batch" | harness binary | Confirms the DSD driver ran a batch; details carry the attempted-line count and socket path. | | ||
| | `test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs:53` | `assert_sometimes!` | "workload delivered at least one dogstatsd line" (`attempted > 0`) | harness binary | Anti-vacuity anchor: a batch can sample count == 0, so "ran" does not imply "sent"; this proves a timeline sometimes actually delivers a line, else delivery checks are vacuous. | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces DogStatD load generation. If you're familiar with
my work in lading you might notice this is very different. The
implementation here is responsible for giving Antithesis choices between
'compliant' dogstatsd lines and 'feral' lines which are mechanically
allowable lines but are in practice not sensible. Also of interest
control of contexts and send rates are jettisoned entirely, that's an
antithesis scheduling concern.
Metrics, events and service-checks are all present. Depending on the
antithesis shots I may need to adjust the ratios of compliant to feral
outcomes.
Change Type
How did you test this PR?
Antithesis shots confirmed load generation behaves like a feral animal.
References