Skip to content

fix(profiler): lock-free class/endpoint/context maps via TripleBufferedDictionary#524

Draft
jbachorik wants to merge 1 commit into
muse/sigsegv-in-recordingfrom
muse/crash-sigsegv-in-std-rb-tree-increment-clean
Draft

fix(profiler): lock-free class/endpoint/context maps via TripleBufferedDictionary#524
jbachorik wants to merge 1 commit into
muse/sigsegv-in-recordingfrom
muse/crash-sigsegv-in-std-rb-tree-increment-clean

Conversation

@jbachorik
Copy link
Copy Markdown
Collaborator

@jbachorik jbachorik commented May 12, 2026

What does this PR do?:

Replaces the SpinLock-guarded Dictionary instances for _class_map, _string_label_map, and _context_value_map with a new TripleBufferedDictionary that eliminates all locking from the read/write fast paths.

TripleBufferedDictionary holds three Dictionary buffers cycling through three roles via a generic TripleBufferRotator<T> template:

  • active — receives new writes (signal handlers + JNI threads), lock-free via CAS
  • dump — snapshot being read by the dump thread; promoted from old active on rotate()
  • scratch — two rotations behind active; ready to be cleared lock-free

The "scratch" role exists for safe lock-free reclamation: when a buffer enters that role, at least one full dump cycle has elapsed since it was last in the active or dump role. That grace period is much longer than any signal-handler (per-thread-locked, drained by lockAll() around the dump) or JNI-thread (microsecond lookup) can plausibly outlive a stale active pointer, so the buffer can be freed without any explicit drain.

bounded_lookup(size_limit=0) is signal-safe (no malloc) and checks the active buffer only — there is no fallback to older snapshots.

As part of this change the following dead code is removed:

  • _class_map_lock (SpinLock)
  • classMapSharedGuard() / classMapTrySharedGuard() on Profiler
  • tryLockSharedBounded() and BoundedOptionalSharedLockGuard on SpinLock

Motivation:

Three production crashes (fingerprint v10.DAECC680F0728EAB44F26DB0B91B703F, 2026-05-06 to 2026-05-08) showed SIGSEGV in std::_Rb_tree_increment via Recording::writeCpoolRecording::writeClassesDictionary::collect, caused by a race between writeClasses and concurrent Dictionary::clear().

PR #516 patched this with a shared-lock, but that introduced tryLockSharedBounded(5) in the signal-handler path (walkVM). Under heavy 100 µs wall-clock load on aarch64 the bounded CAS retries were consistently exhausted, causing class lookups to return -1 and corrupting JFR recordings.

This PR also fixes a related counter-tracking gap: dictionary_classes_keys was always 0 during wall-clock profiling because fill-path inserts went to a buffer with counter id=0. All three buffers now carry the real id.

Note: walkVM's vtable-stub class resolution remains best-effort (it can only find classes that some other path has already inserted into the active buffer); a proper fix would require pre-populating the dictionary via JVMTI ClassPrepare, which is left to a follow-up.

Supersedes PR #522 (fix(profiler): fix SIGSEGV in Dictionary::clear under concurrent lookup).

Additional Notes:

  • The grace period for safe lock-free clearing is the time between two consecutive clearStandby() calls (one full dump interval, typically ≥60s). This is many orders of magnitude longer than any signal-handler or JNI lookup, so explicit drains (RefCountGuard, lockAll) are unnecessary for the dictionary clear path.
  • At most two non-empty buffers exist at any time (active + dump).

How to test the change?:

  • ddprof-lib:gtestDebug_dictionary_ut — covers TripleBufferedDictionary rotation, counter semantics, and concurrent writer safety.
  • DictionaryRotationTest (Java) — counter reset after clearStandby; correct counts after fill-path inserts.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

  • This PR doesn't touch any of that.

  • JIRA: [JIRA-XXXX]

@dd-octo-sts
Copy link
Copy Markdown
Contributor

dd-octo-sts Bot commented May 12, 2026

CI Test Results

Run: #25874376313 | Commit: 026e087 | Duration: 30m 25s (longest job)

All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - - -
8-ibm - - -
8-j9 - -
8-librca - -
8-orcl - - -
11 - - -
11-j9 - -
11-librca - -
17 - -
17-graal - -
17-j9 - -
17-librca - -
21 - -
21-graal - -
21-librca - -
25 - -
25-graal - -
25-librca - -

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-05-14 17:50:54 UTC

@jbachorik jbachorik force-pushed the muse/crash-sigsegv-in-std-rb-tree-increment-clean branch 2 times, most recently from 2f23bab to 132b472 Compare May 13, 2026 07:42
@jbachorik jbachorik changed the title fix(profiler): lock-free class/endpoint/context maps via DoubleBufferedDictionary fix(profiler): lock-free class/endpoint/context maps via TripleBufferedDictionary May 13, 2026
@jbachorik
Copy link
Copy Markdown
Collaborator Author

Reorganization plan

This PR is the foundation of the lock-free dictionary refactor (PR A in the reorganized sequence). It depends on #510 (PR B) merging first to avoid intermittent musl/aarch64 CI failures.

After #510 merges, this PR should be rebased onto main with two specific cleanups:

  1. Drop libraryPatcher_linux.cpp changes from this PR. This PR currently uses a try/catch (abi::__forced_unwind&) approach which is buggy. fix(profiler): line-number-table UAF, musl/aarch64 wrapper canary, and SIGVTALRM teardown race #510 ships the correct fix (no_stack_protector + pthread_exit() + TLS-based cleanup); keep fix(profiler): line-number-table UAF, musl/aarch64 wrapper canary, and SIGVTALRM teardown race #510's version.
  2. Reconcile the duplicate PROF-14545 fix in flightRecorder.cpp/h and the Java memleak tests. The changes are identical between this PR and fix(profiler): line-number-table UAF, musl/aarch64 wrapper canary, and SIGVTALRM teardown race #510, so the rebase should merge cleanly. If conflicts arise, prefer fix(profiler): line-number-table UAF, musl/aarch64 wrapper canary, and SIGVTALRM teardown race #510's version (already reviewed/CI-tested as part of PR B).

After this PR merges, #527 will need to be rebased on top to switch from _class_map_lock + _class_map.lookup() to the TripleBufferedDictionary API.

@jbachorik jbachorik force-pushed the muse/crash-sigsegv-in-std-rb-tree-increment-clean branch from 7844134 to b90761e Compare May 13, 2026 16:15
@jbachorik jbachorik changed the base branch from main to muse/sigsegv-in-recording May 13, 2026 16:15
@jbachorik
Copy link
Copy Markdown
Collaborator Author

Rebased on top of #510 (muse/sigsegv-in-recording). Two changes vs. the previous version:

The unique content of this PR is the TripleBufferedDictionary refactor itself plus its tests. spinlock_bounded_ut.cpp and dictionary_concurrent_ut.cpp are deleted (subsumed by dictionary_ut.cpp).

@jbachorik jbachorik force-pushed the muse/crash-sigsegv-in-std-rb-tree-increment-clean branch from b90761e to 76d919d Compare May 13, 2026 16:52
…ufferedDictionary

Replaces the SpinLock-guarded Dictionary instances for _class_map,
_string_label_map, and _context_value_map with a new TripleBufferedDictionary
that eliminates all locking from the read/write fast paths.

TripleBufferedDictionary holds three Dictionary buffers cycling through three
roles via a generic TripleBufferRotator<T> template:
- active   — receives new writes (signal handlers + JNI threads), lock-free via CAS
- dump     — snapshot being read by the dump thread; promoted from old active on rotate()
- scratch  — two rotations behind active; ready to be cleared lock-free

The scratch role exists for safe lock-free reclamation: when a buffer enters
that role, at least one full dump cycle has elapsed since it was last in the
active or dump role.  That grace period is much longer than any signal-handler
or JNI-thread can plausibly outlive a stale active pointer, so the buffer can
be freed without any explicit drain.

bounded_lookup(size_limit=0) is signal-safe (no malloc) and checks the active
buffer only — no fallback to older snapshots.

Dead code removed:
- _class_map_lock (SpinLock)
- classMapSharedGuard() / classMapTrySharedGuard() on Profiler
- tryLockSharedBounded() / BoundedOptionalSharedLockGuard on SpinLock
- spinlock_bounded_ut.cpp / dictionary_concurrent_ut.cpp (subsumed by dictionary_ut.cpp)

Motivation: three production crashes (fingerprint v10.DAECC680F0728EAB44F26DB0B91B703F)
showed SIGSEGV in std::_Rb_tree_increment via writeCpool → writeClasses →
Dictionary::collect, caused by a race between writeClasses and concurrent
Dictionary::clear().  PR #516 patched it with a shared-lock that exhausted
bounded CAS retries under heavy 100 µs wall-clock load on aarch64, causing
class lookups to return -1 and corrupting JFR recordings.  This change
eliminates the lock entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jbachorik jbachorik force-pushed the muse/crash-sigsegv-in-std-rb-tree-increment-clean branch from 76d919d to 2105b61 Compare May 14, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant