Don't allow simultaneous memory and CPU profiling#3922
Merged
Conversation
quodlibetor
reviewed
Aug 12, 2020
Comment on lines
+77
to
+79
| // SAFETY: We ensure above that memory profiling is off. | ||
| // Since we hold the mutex, nobody else can be turning it back on in the intervening time. | ||
| let stacks = unsafe { prof_time(Duration::from_secs(10), 99, merge_threads) }.await?; |
Contributor
There was a problem hiding this comment.
it might be worth adding an explicit drop(_ctl_lock); after this to ensure that we maintain ownership of the lock in the face of refactorings.
quodlibetor
approved these changes
Aug 12, 2020
benesch
approved these changes
Aug 12, 2020
SangJunBak
added a commit
to SangJunBak/materialize
that referenced
this pull request
Jul 21, 2026
`/cpu` turns on CPU profiling. Before, users could only turn it on via a webpage. We also ensure CPU and memory profiling cannot be turned on at the same time. This is because memory profiling and CPU profiling at the same time can cause deadlocks (see MaterializeInc#3922 for more context). Thus before we start CPU profiling, we turn off memory profilng, then create a drop guard for the remainder of the CPU profiling execution that ensures memory profiling is turned back to its previous state after a drop. `/mode` returns what profiling is enabled and allows users to turn on/off memory profiling.
This was referenced Jul 21, 2026
SangJunBak
added a commit
that referenced
this pull request
Jul 23, 2026
**Extend profiling APIs with CPU profiling support and wire it into mz-debug** **Extend the profiling HTTP APIs with /cpu and /mode routes.** - /cpu turns on CPU profiling. Before, users could only turn it on via a webpage. CPU and memory profiling cannot be turned on at the same time, because doing so can cause deadlocks (see #3922 for more context). Before starting CPU profiling, we turn off memory profiling, then create a drop guard for the remainder of the CPU profiling execution that ensures memory profiling is turned back to its previous state after a drop. - /mode returns what profiling is enabled and allows users to turn memory profiling on/off. **Add a test asserting that CPU profiling disables memory profiling.** Implement CPU profiling in mz-debug: using the new APIs, scrape CPU profiles from each service, also logging if memory profiling is still disabled afterwards. This shouldn't be the case usually, but can happen if for whatever reason, activate on jemalloc fails. **Fix an underflow bug found while testing that caused a panic.** **Update the regression test.** Reuse endpoint functions for the emulator: make the code more consistent by reusing what we already use to branch clusterd and environmentd services for the emulator. ### Motivation For many self managed customers, we've had to manually tell users to go to some webpage to scrape CPU profiles which is too much TOIL. This feature now also gets CPU profiles along with memory profiles, default to 10 seconds. ### Verification Created some unit tests and manually tested via my personal kind environment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See tikv/pprof-rs#36. Running the pprof CPU profiler at the same time as the jemalloc profiler is apparently UB. It reliably causes deadlocks in Materialize if the jemalloc sampling rate is aggressive enough (e.g. 2^14).
If another user accesses
/profwhile a CPU profile is running, this will have the unfortunate side effect of hanging their page load until the profile is done. We can improve that UX as a follow-up, but it requires a bit of refactoring (it's not as simple as just callingtry_lock) and since it's a purely cosmetic issue, I didn't want it to block this UB safety issue.This change is