Skip to content

🍒 12013 - Fix Metaspace leak in SCA Reachability retransform and opcode injection#12075

Merged
claponcet merged 1 commit into
release/v1.64.xfrom
clara.poncet/backport-pr-12013
Jul 27, 2026
Merged

🍒 12013 - Fix Metaspace leak in SCA Reachability retransform and opcode injection#12075
claponcet merged 1 commit into
release/v1.64.xfrom
clara.poncet/backport-pr-12013

Conversation

@claponcet

Copy link
Copy Markdown
Contributor

Backport #12013 to release/v1.64.x

…on (#12013)

Fix Metaspace leak in SCA Reachability (APPSEC-69201)

- Bound retransformClasses() retries: track per-class failure counts and give
  up after MAX_RETRANSFORM_ATTEMPTS instead of re-queueing failed classes
  forever, which retained Class<?>/ClassLoader references indefinitely.
- Cover all 13 ASM MethodVisitor visitXxxInsn callbacks in
  ScaMethodCallbackInjector.MethodEntryInjector so the entry-point injection
  triggers on the first bytecode instruction regardless of its opcode kind,
  instead of only on the 4 previously handled ones.

Bisect failing retransform batches instead of coupling them by failure count

Split pendingRetransform into independent per-batch queue entries and retransform each
batch separately; on failure, bisect multi-class batches into two halves deferred to the
next heartbeat instead of tying an unrelated healthy class to a permanently-failing one.

Simplify singleton-batch retransform retry to drop-on-first-failure

Replace the retransformFailureCount/MAX_RETRANSFORM_ATTEMPTS bounded
retry with an immediate drop once a batch is narrowed down to a
single class, per the performance-over-detection trade-off agreed
with mcculls on PR #12013.

Batch already-loaded classes as one shared retransform group

checkAlreadyLoadedClasses() previously queued each already-loaded
vulnerable class as its own singleton batch, turning the common
all-succeed startup path into one retransformClasses() call per
class instead of one call for all of them. Queue them as a single
shared batch instead; bisection still applies on the next heartbeat
if that batch fails.

Found by Codex review on PR #12013.

Add test coverage for remaining MethodEntryInjector opcode overrides

jacocoTestCoverageVerification failed: visitIntInsn/visitJumpInsn/visitIincInsn/
visitTableSwitchInsn/visitLookupSwitchInsn/visitMultiANewArrayInsn were never
exercised by any test, leaving instruction coverage at 0.7 against the 0.8
minimum for the appsec module.

Remove non-modifiable classes from batch in place instead of copying to a second list

Addresses mccull's review comment: mutate the polled batch directly via
removeIf() instead of allocating a second ArrayList to hold the modifiable
classes. Updates tests that seeded pendingRetransform with immutable
singletonList() batches, which is no longer valid now that batches must
support in-place removal.

Merge branch 'master' into sca-bug

Merge branch 'master' into sca-bug

Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
(cherry picked from commit 428bb15)
@claponcet
claponcet requested a review from a team as a code owner July 27, 2026 07:48
@claponcet claponcet added type: bug fix Bug fix comp: asm waf Application Security Management (WAF) labels Jul 27, 2026

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The changed retransformation batching and ASM entry injection paths were reviewed against the PR diff and surrounding tests, with no clear diff-only behavioral regression identified. Focused Gradle execution was unavailable because the wrapper could not download offline and the installed build requires Java 25; no additional tests were recommended because the PR already adds coverage for the relevant opcode and failure-batching cases.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit b06c87e · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 27, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 89.47%
Overall Coverage: 54.16% (-2.76%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b06c87e | Docs | Datadog PR Page | Give us feedback!

@jandro996 jandro996 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thnks!!

@dd-octo-sts

dd-octo-sts Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.67 s 14.65 s [-0.8%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 13.64 s 13.73 s [-1.6%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.45 s 17.31 s [+0.0%; +1.6%] (maybe worse)
startup:petclinic:iast:Agent 17.26 s 17.51 s [-2.2%; -0.6%] (maybe better)
startup:petclinic:profiling:Agent 16.70 s 17.37 s [-8.1%; +0.4%] (no difference)
startup:petclinic:sca:Agent 17.42 s 17.38 s [-0.6%; +1.1%] (no difference)
startup:petclinic:tracing:Agent 16.14 s 16.20 s [-6.3%; +5.5%] (unstable)

Commit: b06c87e3 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@claponcet
claponcet merged commit ca39cc8 into release/v1.64.x Jul 27, 2026
779 of 786 checks passed
@claponcet
claponcet deleted the clara.poncet/backport-pr-12013 branch July 27, 2026 08:58
@github-actions github-actions Bot added this to the 1.64.2 milestone Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: asm waf Application Security Management (WAF) type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants