fix(tsan): clean TSan gtest suite; fix races and RefCountGuard activation window

[jbachorik]

What does this PR do?:

Makes the TSan gtest suite pass cleanly on aarch64 (Docker Desktop / linuxkit kernel with 48-bit VMA), fixes two real data races plus a correctness bug in RefCountGuard that TSan exposed, and closes two ASan-reported issues (bounds-annotation gap and a test-fixture memory leak).

Motivation:

TSan gtests were crashing before any test ran due to a VMA mismatch between GCC 11's libtsan (39-bit VMA only) and Docker Desktop's linuxkit kernel (48-bit VMA). Once the environment was fixed, TSan found two real races and revealed the RefCountGuard activation-window gap. ASan found a missing no_sanitize(bounds) on non-x86 architectures and a ProfiledThread leak in the T-02 test fixture. All four sanitizer jobs are now blocking — pipeline fails on any finding.

Additional Notes:

Races fixed (found by TSan):

• callTraceHashTable.cpp: plain read of _table in put() raced with the atomic CAS in the table-expansion path. Fixed with __atomic_load_n(..., __ATOMIC_ACQUIRE) to pair with the ACQ_REL CAS.

• stringDictionary.h: plain read of row->next before the overflow-chain CAS raced with another thread's CAS. Fixed with __atomic_load_n(..., __ATOMIC_RELAXED) (the ACQUIRE load at the CAS site already provides the required happens-before for the newly-allocated SBTable).

• waitForRefCountToClear / waitForAllRefCountsToClear skipped slots with count==0 without checking active_ptr. The pointer-first activation protocol stores active_ptr (RELEASE) before the count increment (RELEASE), so a slot can be live (active_ptr set) while count is still zero. The scanner now checks active_ptr on zero-count slots to close this window. refCountGuard_ut.cpp exercises this scenario.

ASan fixes:

• no_sanitize("bounds") extended to all architectures (was x86-only); the annotation guards an intentional out-of-bounds read used to detect array-vs-pointer calling conventions in the JVM.

• T-02 ProfiledThread leak: clearCurrentThreadTLS() now returns the detached pointer instead of void, and a companion deleteForTest() helper (destructor is private) lets the test complete the simulated teardown. Previously, the object leaked because release() found TLS already null and did nothing, and the pthread key destructor was never called for a null value.

TSan infrastructure (aarch64 / Docker Desktop):

• run-docker-tests.sh: installs clang-17 + libclang-rt-17-dev on aarch64 glibc images (GCC 11's TSan doesn't support 48-bit VMA); adds --privileged + sysctl vm.mmap_rnd_bits=28 so TSan can map its shadow; passes -Pnative.forceCompiler=clang++-17.
• PlatformUtils.kt: locateLibtsan() now prefers clang's own libclang_rt.tsan-<arch>.so over GCC's libtsan, matching the existing locateLibasan() pattern.
• ConfigurationPresets.kt: uses dynamic lib name in TSan linker args; adds JVM-safe TSAN_OPTIONS (handle_segv/sigbus=0, use_sigaltstack=0, io_sync=0); sets GTEST_DEATH_TEST_STYLE=threadsafe (fork is unsafe under TSan).
• GtestTaskBuilder.kt: strips -ltsan/-lclang_rt.tsan* from gtest link so clang statically embeds its own TSan runtime, preventing GCC/clang runtime collision.
• ProfilerTestPlugin.kt: skips Java integration tests under TSan — the profiler's SIGPROF handler fires at 1ms inside TSan-instrumented code, causing re-entrance crashes. TSan coverage is provided by the C++ gtest suite (consistent with CI).
• tsan.supp: suppresses JVM-internal libnio.so/libjava.so FD-management races.

CI:

• All four sanitizer jobs (ASan amd64/arm64, TSan amd64/arm64) are now blocking — allow_failure: true removed. Pipeline fails on any leak or data race.

How to test the change?:

./utils/run-docker-tests.sh --config=tsan --gtest

Expected: BUILD SUCCESSFUL, all 31 gtestTsan_* tasks pass, testTsan SKIPPED.

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.
• JIRA: [JIRA-XXXX]

[jbachorik]

@codex review

[Copilot]

Pull request overview

This PR fixes TSan-detected races and updates the TSan test/build path, especially for aarch64 Docker Desktop environments with 48-bit VMA support needs.

Changes:

• Fixes atomic access races in CallTraceHashTable and StringDictionary.
• Closes the RefCountGuard activation-window gap and adds regression coverage.
• Updates TSan Docker/build/test configuration to use compatible clang runtimes and skip incompatible JVM integration tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/run-docker-tests.sh	Adds aarch64 clang-17 setup and privileged/sysctl handling for TSan.
gradle/sanitizers/tsan.supp	Adds JVM library race suppressions.
ddprof-lib/src/test/cpp/refCountGuard_ut.cpp	Adds activation-window regression tests.
ddprof-lib/src/main/cpp/stringDictionary.h	Uses an atomic relaxed load for overflow-chain hinting.
ddprof-lib/src/main/cpp/refCountGuard.h	Updates protocol documentation for active pointer scanning.
ddprof-lib/src/main/cpp/refCountGuard.cpp	Checks active_ptr during zero-count wait scans.
ddprof-lib/src/main/cpp/callTraceHashTable.cpp	Uses acquire load for _table in put().
build-logic/conventions/src/main/kotlin/com/datadoghq/profiler/ProfilerTestPlugin.kt	Skips JVM integration tests under TSan.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/util/PlatformUtils.kt	Prefers clang TSan runtime when using clang.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/gtest/GtestTaskBuilder.kt	Strips explicit TSan runtime link flags for gtest executables.
build-logic/conventions/src/main/kotlin/com/datadoghq/native/config/ConfigurationPresets.kt	Updates TSan linker args and test environment options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2560589453

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 21e76a33c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[dd-octo-sts]

CI Test Results

Run: #26598758953 | Commit: 6d5ee81 | Duration: 12m 23s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK	glibc-aarch64/debug	glibc-amd64/debug	musl-aarch64/debug	musl-amd64/debug
8	-	✅	-	-
8-ibm	-	✅	-	-
8-j9	✅	✅	-	-
8-librca	-	-	✅	✅
8-orcl	-	✅	-	-
11	-	✅	-	-
11-j9	✅	✅	-	-
11-librca	-	-	✅	✅
17	✅	✅	-	-
17-graal	✅	✅	-	-
17-j9	✅	✅	-	-
17-librca	-	-	✅	✅
21	✅	✅	-	-
21-graal	✅	✅	-	-
21-librca	-	-	✅	✅
25	✅	✅	-	-
25-graal	✅	✅	-	-
25-librca	-	-	✅	✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0

---

Updated: 2026-05-28 20:13:03 UTC