Stop retrying aborted Karate scenarios by daniel-mohedano · Pull Request #12015 · DataDog/dd-trace-java

daniel-mohedano · 2026-07-21T07:55:00Z

What Does This Do

Traces whether a Karate 2 scenario emitted SCENARIO_ENTER. Scenarios that return before entering are not finished or retried.

Motivation

With abortSuiteOnFailure enabled, ScenarioRuntime.call() can return before the scenario starts. The retry advice previously attempted to finish a test that did not exist.

Since the execution policy was not advanced, it remained applicable and the retry loop could repeatedly execute the aborted scenario without terminating.

Additional Notes

Includes regression coverage for an aborted suite during retry handling.

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

cit-pr-commenter-54b7da · 2026-07-21T08:06:50Z

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.33	55.43	$\color{red}{\blacktriangle}$ +0.90	55.43	$\color{red}{\blacktriangle}$ +0.90	72/268
agentEvpProxy	57.58	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:08:15Z

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	89.16	87.80	$\color{red}{\blacktriangle}$ +1.36	87.80	$\color{red}{\blacktriangle}$ +1.36	37/141
agentless	81.72	81.05	$\color{red}{\blacktriangle}$ +0.67	81.05	$\color{red}{\blacktriangle}$ +0.67	34/139
agentlessCodeCoverage	96.19	97.04	$\color{green}{\blacktriangledown}$ -0.85	97.04	$\color{green}{\blacktriangledown}$ -0.85	34/137
agentlessLineCoverage	112.44	111.62	$\color{red}{\blacktriangle}$ +0.82	111.62	$\color{red}{\blacktriangle}$ +0.82	33/136

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

datadog-datadog-prod-us1-2 · 2026-07-21T08:08:35Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 69.73% (+12.18%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0799b42 | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-07-21T08:10:04Z

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	37.77	36.42	$\color{red}{\blacktriangle}$ +1.35	36.42	$\color{red}{\blacktriangle}$ +1.35	36/136
agentless	37.61	36.42	$\color{red}{\blacktriangle}$ +1.19	36.42	$\color{red}{\blacktriangle}$ +1.19	36/136
agentlessCodeCoverage	44.94	44.48	$\color{red}{\blacktriangle}$ +0.46	44.48	$\color{red}{\blacktriangle}$ +0.46	36/136
agentlessLineCoverage	56.48	73.34	$\color{green}{\blacktriangledown}$ -16.86	74.82	$\color{green}{\blacktriangledown}$ -18.34	35/136

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:11:40Z

Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	4.92	9.54	$\color{green}{\blacktriangledown}$ -4.62	10.33	$\color{green}{\blacktriangledown}$ -5.41	37/128
agentless	10.77	9.73	$\color{red}{\blacktriangle}$ +1.04	10.13	$\color{red}{\blacktriangle}$ +0.64	37/128
agentlessCodeCoverage	17.78	15.72	$\color{red}{\blacktriangle}$ +2.06	17.03	$\color{red}{\blacktriangle}$ +0.75	36/126

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:12:07Z

Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.90	9.73	$\color{red}{\blacktriangle}$ +2.17	9.54	$\color{red}{\blacktriangle}$ +2.36	32/82

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:13:08Z

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	17.98	21.65	$\color{green}{\blacktriangledown}$ -3.67	21.65	$\color{green}{\blacktriangledown}$ -3.67	40/144
agentless	14.74	18.45	$\color{green}{\blacktriangledown}$ -3.71	18.82	$\color{green}{\blacktriangledown}$ -4.08	38/139
agentlessCodeCoverage	15.77	19.99	$\color{green}{\blacktriangledown}$ -4.22	19.99	$\color{green}{\blacktriangledown}$ -4.22	36/135
agentlessLineCoverage	23.31	29.23	$\color{green}{\blacktriangledown}$ -5.92	29.82	$\color{green}{\blacktriangledown}$ -6.51	36/135

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:15:31Z

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.10	13.40	$\color{green}{\blacktriangledown}$ -1.30	13.13	$\color{green}{\blacktriangledown}$ -1.03	35/132
agentless	10.93	12.12	$\color{green}{\blacktriangledown}$ -1.19	12.12	$\color{green}{\blacktriangledown}$ -1.19	33/127
agentlessCodeCoverage	13.78	15.41	$\color{green}{\blacktriangledown}$ -1.63	15.11	$\color{green}{\blacktriangledown}$ -1.33	33/127
agentlessLineCoverage	17.25	18.45	$\color{green}{\blacktriangledown}$ -1.20	18.82	$\color{green}{\blacktriangledown}$ -1.57	32/129

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

dd-octo-sts · 2026-07-21T08:19:14Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.94 s	13.94 s	[-0.7%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent	12.98 s	13.02 s	[-1.1%; +0.4%] (no difference)
startup:petclinic:appsec:Agent	17.02 s	16.76 s	[+0.3%; +2.8%] (maybe worse)
startup:petclinic:iast:Agent	16.98 s	16.95 s	[-0.8%; +1.1%] (no difference)
startup:petclinic:profiling:Agent	16.83 s	16.79 s	[-1.0%; +1.5%] (no difference)
startup:petclinic:sca:Agent	17.02 s	16.54 s	[-1.4%; +7.3%] (no difference)
startup:petclinic:tracing:Agent	16.11 s	16.17 s	[-1.5%; +0.7%] (no difference)

Commit: 0799b42c · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

cit-pr-commenter-54b7da · 2026-07-21T08:19:51Z

Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	93.31	93.23	$\color{red}{\blacktriangle}$ +0.08	93.23	$\color{red}{\blacktriangle}$ +0.08	37/145
agentless	89.53	89.58	$\color{green}{\blacktriangledown}$ -0.05	89.58	$\color{green}{\blacktriangledown}$ -0.05	37/143
agentlessCodeCoverage	98.31	99.00	$\color{green}{\blacktriangledown}$ -0.69	99.00	$\color{green}{\blacktriangledown}$ -0.69	35/140
agentlessLineCoverage	100.37	99.00	$\color{red}{\blacktriangle}$ +1.37	99.00	$\color{red}{\blacktriangle}$ +1.37	34/137

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:24:06Z

Test Environment - okhttp

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:34:19Z

Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.24	16.36	$\color{green}{\blacktriangledown}$ -0.12	16.36	$\color{green}{\blacktriangledown}$ -0.12	37/132
agentless	10.38	9.73	$\color{red}{\blacktriangle}$ +0.65	9.73	$\color{red}{\blacktriangle}$ +0.65	36/132
agentlessCodeCoverage	13.55	13.40	$\color{red}{\blacktriangle}$ +0.15	13.40	$\color{red}{\blacktriangle}$ +0.15	36/132
agentlessLineCoverage	23.05	32.30	$\color{green}{\blacktriangledown}$ -9.25	32.95	$\color{green}{\blacktriangledown}$ -9.90	35/132

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

cit-pr-commenter-54b7da · 2026-07-21T08:50:30Z

Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	-4.12	11.42	$\color{green}{\blacktriangledown}$ -15.54	12.62	$\color{green}{\blacktriangledown}$ -16.74	37/137
agentless	26.07	20.39	$\color{red}{\blacktriangle}$ +5.68	17.03	$\color{red}{\blacktriangle}$ +9.04	37/136
agentlessCodeCoverage	56.77	87.80	$\color{green}{\blacktriangledown}$ -31.03	84.36	$\color{green}{\blacktriangledown}$ -27.59	37/136
agentlessLineCoverage	125.48	136.34	$\color{green}{\blacktriangledown}$ -10.86	136.34	$\color{green}{\blacktriangledown}$ -10.86	36/135

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

… daniel.mohedano/karate-finish-after-post-processing

…to daniel.mohedano/karate-stop-aborted-retries

datadog-datadog-prod-us1-2

More details

The PR introduces sound defensive tracking to prevent infinite retry loops when scenarios abort before starting. By recording whether SCENARIO_ENTER fires, the code now correctly skips retry processing for aborted scenarios where the execution policy wouldn't advance. The fix is minimal, targeted, and preserves existing behavior for normal scenarios while fixing the infinite loop on abort.

Was this helpful? React 👍 or 👎

_{📊 Validated against 4 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 2f85786 · What is Autotest? · Any feedback? Reach out in #autotest}

# Conflicts: # dd-java-agent/instrumentation/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateScenarioAdvice.java

daniel-mohedano · 2026-07-24T08:59:26Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-24T08:59:30Z

View all feedbacks in Devflow UI.

2026-07-24 08:59:30 UTC ℹ️ Start processing command /merge

2026-07-24 08:59:34 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-24 09:59:33 UTC ❌ MergeQueue: This merge request is not mergeable, blocked by github

PR can't be merged according to github policy

pr-shepherd-6ad11f · 2026-07-24T09:00:11Z

PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the pr-shepherd:ignore label.

If you have any questions, reach the team in #ai-devx-flow.

PerfectSlayer · 2026-07-27T06:46:43Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-27T06:46:47Z

View all feedbacks in Devflow UI.

2026-07-27 06:46:46 UTC ℹ️ Start processing command /merge

2026-07-27 06:46:51 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-27 07:38:01 UTC ℹ️ MergeQueue: This merge request was merged

daniel-mohedano added 3 commits July 21, 2026 09:31

fix: avoid retrying untracked karate scenarios

3a1eec0

fix: finish karate tests after scenario post-processing

43d9442

fix: stop retrying aborted karate scenarios

60a3dd7

daniel-mohedano added type: bug fix Bug fix comp: ci visibility Continuous Integration Visibility labels Jul 21, 2026

daniel-mohedano added 6 commits July 21, 2026 11:30

fix: merge test scenarios

42b9368

Merge branch 'daniel.mohedano/karate-ignore-untracked-scenarios' into…

d0a8d0a

… daniel.mohedano/karate-finish-after-post-processing

Merge branch 'daniel.mohedano/karate-finish-after-post-processing' in…

c0bb9f8

…to daniel.mohedano/karate-stop-aborted-retries

fix: honor expected karate failures

5622501

Merge branch 'daniel.mohedano/karate-finish-after-post-processing' in…

3929436

…to daniel.mohedano/karate-stop-aborted-retries

nit: comment

2f85786

daniel-mohedano marked this pull request as ready for review July 21, 2026 10:56

daniel-mohedano requested a review from a team as a code owner July 21, 2026 10:56

datadog-datadog-prod-us1-2 Bot reviewed Jul 21, 2026

View reviewed changes

daniel-mohedano changed the title ~~Stop retrying aborted karate scenarios~~ Stop retrying aborted Karate scenarios Jul 21, 2026

juan-fernandez approved these changes Jul 21, 2026

View reviewed changes

Base automatically changed from daniel.mohedano/karate-finish-after-post-processing to master July 23, 2026 14:11

Merge branch 'master' into daniel.mohedano/karate-stop-aborted-retries

0799b42

# Conflicts: # dd-java-agent/instrumentation/karate/karate-2.0/src/main/java21/datadog/trace/instrumentation/karate2/KarateScenarioAdvice.java

pr-shepherd-6ad11f Bot added the pr-shepherd:in_progress label Jul 24, 2026

DataDog deleted a comment from dd-octo-sts Bot Jul 27, 2026

PerfectSlayer added type: bug fix Bug fix and removed pr-shepherd:in_progress type: bug fix Bug fix labels Jul 27, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit 77f9ba5 into master Jul 27, 2026
786 of 791 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the daniel.mohedano/karate-stop-aborted-retries branch July 27, 2026 07:38

github-actions Bot added this to the 1.65.0 milestone Jul 27, 2026

Uh oh!

Conversation

daniel-mohedano commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sbt-scalatest

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - netflix-zuul

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - nebula-release-plugin

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - pass4s

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - heliboard

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - reactive-streams-jvm

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-kotlin

Uh oh!

dd-octo-sts Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - jolokia

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - okhttp

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - spring_boot

Uh oh!

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Environment - sonar-java

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

daniel-mohedano commented Jul 24, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-shepherd-6ad11f Bot commented Jul 24, 2026

Uh oh!

PerfectSlayer commented Jul 27, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

daniel-mohedano commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Jul 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 24, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 27, 2026 •

edited

Loading