test(telemetry): stop transient fork children flaking test_app_heartbeats_delays#7296
Merged
Merged
Conversation
bm1549
added
the
ai-generated
Contributor
|
|
… check test_app_heartbeats_delays buckets every app-heartbeat in the scenario by runtime_id and asserts each measured runtime's average inter-heartbeat delay stays within [0.75x, 1.5x] of the interval. The session-id tests (test_session_id_headers_across_forks/_across_spawned) spawn short-lived children via /spawn_child whose telemetry lands in the same shared pool; on uwsgi-poc a forked child emits a heartbeat under its parent's runtime_id before regenerating its own, producing a transient runtime with a sub-interval "duplicate" (~0.05s) whose average (~1.1s) trips "too fast". Select measurable runtimes by lifespan (first->last heartbeat) rather than heartbeat count, so transient ~2s children are excluded while a genuinely slow-drifting worker -- which emits fewer heartbeats over the same window -- stays measured and its drift is still caught. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bm1549
force-pushed
the
brian.marks/fix-heartbeat-drift-fork-isolation
branch
from
fccde0d to
bd9305e
Compare
🎉 All green!🧪 All tests passed 🔗 Commit SHA: bd9305e | Docs | Datadog PR Page | Give us feedback! |
bm1549
requested review from
cyrille-leclerc and
pawelchcki
and removed request for
a team
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd9305e95e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mabdinur
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
test_app_heartbeats_delaysis flaky only onuwsgi-poc(43 failures onmainin the last 30 days, while every other variant passes), always failingHeartbeat sent too fast: ~1.1s.The test buckets every
app-heartbeatbyruntime_idand checks each runtime's average inter-heartbeat delay against[0.75x, 1.5x]of the interval.uwsgi-pocis the only weblog running a prefork uwsgi master + worker with ddtrace bootstrapped at the uwsgi level, so anos.fork()from the sibling session-id tests (/spawn_child) leaves a short-lived extra runtime in the shared telemetry pool: the forked child emits a heartbeat under its parent'sruntime_idbefore regenerating its own, so that runtime has two heartbeats ~0.05s apart. Its average (~1.1s) trips "too fast". The main app worker is healthy (~2s cadence); single-process weblogs likeflask-pocnever produce this transient runtime.Changes
_get_heartbeat_delays_by_runtimenow selects measurable runtimes by lifespan (first→last heartbeat) instead of heartbeat count. The transient fork child lives ~2s and is excluded; long-lived application runtimes span the observation window and are measured. Lifespan is cadence-independent on purpose: a genuinely slow-drifting worker emits fewer heartbeats but still spans the window, so its drift is still caught (a count-based filter would wrongly drop it).Workflow
🚀 Once your PR is reviewed and the CI green, you can merge it!
:lifebuoy: #apm-shared-testing :lifebuoy:
Reviewer checklist
tests/ormanifests/is modified ? I have the approval from R&P team — N/A: onlytests/is modified.🤖 Generated with Claude Code