test: stabilize flaky tests by r1viollet · Pull Request #2256 · DataDog/libdatadog

r1viollet · 2026-07-21T08:09:27Z

What does this PR do?

Skips one ptrace thread-enumeration test under Miri.

Makes async test assertions wait for the expected request instead of relying on fixed sleeps.

Clears the macOS target/panic-abort cache before tests as a short-term workaround for stale cached build artifacts.

Motivation

These flaky tests and the macOS cache issue are currently blocking builds.

Additional Notes

The macOS cache cleanup is intended as a short-term unblocker. A separate draft PR moves panic-abort artifacts outside the cached target tree.

How to test the change?

cargo +nightly-2026-02-08 fmt --all -- --check
cargo test -q -p libdd-data-pipeline-ffi exporter_send_telemetry_test
cargo test -q -p libdd-profiling exporter::exporter_manager::tests -- --nocapture
cargo test -q -p libdd-common --features reqwest test_file_dump_captures_http_request -- --nocapture
Linux workspace: cargo test -q -p libdd-data-pipeline-ffi exporter_send_telemetry_test repeated 30 times
Linux workspace: RUSTUP_TOOLCHAIN=nightly-2026-02-08 MIRIFLAGS="-Zmiri-disable-isolation" cargo miri test -p libdd-crashtracker enumerate_discovers_spawned_thread -- --nocapture

r1viollet · 2026-07-21T08:11:45Z

Link to failures:
https://github.com/DataDog/libdatadog/actions/runs/29804761537/job/88553008560
https://github.com/DataDog/libdatadog/actions/runs/29806802452/job/88558985923#step:8:3241

pr-commenter · 2026-07-21T08:21:11Z

Benchmarks

Comparison

Benchmark execution time: 2026-07-21 10:41:53

Comparing candidate commit 059df57 in PR branch r1viollet/stabilize-flaky-tests with baseline commit fb65e46 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 123 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Omitted due to size.

Baseline

Omitted due to size.

datadog-prod-us1-5 · 2026-07-21T08:23:26Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 93.10%
• Overall Coverage: 74.54% (-0.02%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 059df57 | Docs | Datadog PR Page | Give us feedback!}

yannham

If this unblocks, let's merge this 👍

yannham · 2026-07-21T08:39:36Z

    }

    #[test]
+    #[cfg_attr(miri, ignore)]


I'm curious, why would miri behave differently wrt to flakiness? Isn't just a red herring (limiting the probability of hitting the issue if we run the test twice as less? Is it a new test that just fails under Miri consistently?

So I could not repro this. Clanker has theories about the test that could be an issue:

For this specific test:

enumerate_discovers_spawned_thread

the test spawns a thread, receives its TID over a channel, then enumerates /proc/self/task and expects that TID to be present.

The channel guarantees the child thread sent its TID, but it does not necessarily prove that Miri/host /proc view will expose that interpreted thread exactly like a normal native Linux thread at that point. Under Miri, threads are simulated/interpreted, while /proc
is an OS interface. With -Zmiri-disable-isolation, Miri allows more host access, but that does not make every host-observable thread assumption robust.

dd-octo-sts · 2026-07-21T08:43:29Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.13 MB	8.13 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	88.03 MB	88.04 MB	+0% (+1.94 KB) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	99.31 MB	99.32 MB	+0% (+3.34 KB) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	10.86 MB	10.86 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	26.55 MB	26.55 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	191.10 MB	191.09 MB	-0% (-8.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	1.04 GB	1.04 GB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	8.65 MB	8.65 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	89.18 KB	89.18 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	25.31 MB	25.31 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	50.49 MB	50.49 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	23.13 MB	23.13 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	195.80 MB	195.79 MB	-0% (-16.00 KB) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	1.03 GB	1.03 GB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	6.67 MB	6.67 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	90.58 KB	90.58 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	27.22 MB	27.22 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	48.06 MB	48.06 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	78.53 MB	78.53 MB	+0% (+432 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	9.04 MB	9.04 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	94.02 MB	94.02 MB	+0% (+1.00 KB) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.00 MB	11.00 MB	0% (0 B) 👌

r1viollet · 2026-07-21T08:47:30Z

        with:
          cache-targets: true # cache build artifacts
          cache-bin: true # cache the ~/.cargo/bin directory
+      - name: "Clear macOS panic-abort cache"


@gyuheon0h this is a workaround, I think we should not be caching this.

r1viollet · 2026-07-21T08:49:35Z

More failures...
https://github.com/DataDog/libdatadog/actions/runs/29813190571/job/88578758174?pr=2256

r1viollet · 2026-07-21T10:14:49Z

New failure: https://gitlab.ddbuild.io/DataDog/apm-reliability/libddprof-build/-/jobs/1877379843

r1viollet · 2026-07-21T10:15:10Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-21T10:15:14Z

View all feedbacks in Devflow UI.

2026-07-21 10:15:13 UTC ℹ️ Start processing command /merge

2026-07-21 10:15:20 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-21 10:58:12 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in main is approximately 1h (p90).

2026-07-21 11:40:03 UTC ℹ️ MergeQueue: This merge request was merged

test: stabilize flaky tests

b07312e

github-actions Bot added data-pipeline crashtracker labels Jul 21, 2026

r1viollet marked this pull request as ready for review July 21, 2026 08:11

r1viollet requested review from a team as code owners July 21, 2026 08:11

scottgerring approved these changes Jul 21, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: waiting label Jul 21, 2026

yannham approved these changes Jul 21, 2026

View reviewed changes

ci: clear macOS panic-abort target cache

c666899

r1viollet requested a review from a team as a code owner July 21, 2026 08:46

github-actions Bot added the ci-build label Jul 21, 2026

r1viollet commented Jul 21, 2026

View reviewed changes

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: queued mergequeue-status: in_progress mergequeue-status: rejected and removed mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress labels Jul 21, 2026

test: stabilize profiling exporter tests

059df57

r1viollet requested a review from a team as a code owner July 21, 2026 10:11

github-actions Bot added the common label Jul 21, 2026

gh-worker-dd-devflow-36fce6 Bot added mergequeue-status: waiting mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: rejected mergequeue-status: waiting mergequeue-status: queued labels Jul 21, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 054402d into main Jul 21, 2026
132 of 135 checks passed

gh-worker-dd-devflow-36fce6 Bot removed the mergequeue-status: in_progress label Jul 21, 2026

gh-worker-dd-mergequeue-cf854d Bot deleted the r1viollet/stabilize-flaky-tests branch July 21, 2026 11:39

gh-worker-dd-devflow-36fce6 Bot added the mergequeue-status: done label Jul 21, 2026

dd-octo-sts Bot mentioned this pull request Jul 30, 2026

chore(release): proposal for libdd-http-client, libdd-tracer-flare #2299

Closed

Uh oh!

Conversation

r1viollet commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

r1viollet commented Jul 21, 2026

Uh oh!

pr-commenter Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

Candidate

Baseline

Uh oh!

datadog-prod-us1-5 Bot commented Jul 21, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yannham left a comment

Choose a reason for hiding this comment

Uh oh!

yannham Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

r1viollet Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

r1viollet Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

r1viollet commented Jul 21, 2026

Uh oh!

r1viollet commented Jul 21, 2026

Uh oh!

r1viollet commented Jul 21, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r1viollet commented Jul 21, 2026 •

edited

Loading

pr-commenter Bot commented Jul 21, 2026 •

edited

Loading

datadog-prod-us1-5 Bot commented Jul 21, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

dd-octo-sts Bot commented Jul 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 21, 2026 •

edited

Loading