remove flakiness on kafka tests#11972
Conversation
amarziali
added
tag: no release notes
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ff84d2c0e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: 937a1b7 | Docs | Datadog PR Page | Give us feedback! |
![datadog-datadog-us1-prod[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The PR removes waitForTraces(1) from Kafka test onMessage callbacks, which was causing flakiness by blocking the consumer thread while its span was still open (the span doesn't close until the next hasNext() call after onMessage returns, not during it). Removing the barrier is correct and safe for most tests. However, in "test spring kafka template produce and consume" in both kafka-clients-0.11 and kafka-clients-3.8, the same waitForTraces(1) was also the only synchronization ensuring the parent trace landed at TEST_WRITER[0] before the consumer trace; without it the consumer trace can be written first, making the subsequent TEST_WRITER[0][2] access an IndexOutOfBoundsException. Fixed by replacing it with a sorted snapshot using SORT_TRACES_BY_ID, matching the pattern already used in latestDepTest.
📊 Validated against 6 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 2ff84d2 · What is Autotest? · Any feedback? Reach out in #autotest
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
@codex review |
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
DDCI pipeline didn't start (sourcing_failed)... Please retry. DDCI Change Request: |
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
3 participants
What Does This Do
The flaky test was caused by onMessage() calling TEST_WRITER.waitForTraces(1) on the consumer thread. That blocked the same thread that needed to continue and finish the child span. If the root span was finished but the child stayed open for over the 500ms PendingTraceBuffer watchdog window, the trace could be force-flushed early with only the root span. Later, the child span finished and was written separately, producing two 1-span traces instead of one 2-span trace.
The wait was an old leftover used for deterministic ordering, but ordering is no longer needed because the tests now sort traces before asserting.
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]