Fix guest tracing deadlock when exception happens during tracing data serialization #1066
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dblnz
requested review from
devigned,
jprendes,
ludfjig,
marosset,
simongdavies and
syntactically
as code owners
jsturtevant
reviewed
Dec 2, 2025
jsturtevant
reviewed
Dec 2, 2025
ludfjig
previously approved these changes
Dec 2, 2025
Contributor
ludfjig
left a comment
ludfjig
left a comment
There was a problem hiding this comment.
LGTM. Note that the linked issue will be closed when this merges which may or may not be your intention
Contributor
There was a problem hiding this comment.
Pull request overview
This PR addresses a deadlock issue in guest tracing that occurs when exceptions happen during tracing data serialization. The fix converts direct lock() calls to try_lock() throughout the guest tracing subsystem to detect re-entrant access patterns, and removes the #[instrument] macro from out32() to prevent span creation during exception handling.
Key changes:
- Replaced
lock()withtry_lock().expect()in subscriber methods and lib.rs functions to detect and panic on re-entrant access - Removed
#[instrument]fromout32()which can be called from exception contexts - Simplified string creation in visitor.rs (refactoring cleanup)
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| src/hyperlight_guest_tracing/src/subscriber.rs | Converted all Subscriber trait methods to use try_lock() with panic on failure to detect re-entrant access |
| src/hyperlight_guest_tracing/src/lib.rs | Updated set_start_tsc, end_trace, clean_trace_state, and guest_trace_info to use try_lock(); fixed "guset" typo |
| src/hyperlight_guest/src/exit.rs | Removed #[instrument] from out32() and added documentation explaining why tracing is avoided in exception contexts |
| src/hyperlight_guest_tracing/src/visitor.rs | Refactored string creation to use String::from() directly instead of creating empty String and pushing |
dblnz
force-pushed
the
fix-trace-bug-1
branch
2 times, most recently
from
748dd61 to
8045903
Compare
…t to report trace data When guest tracing is enabled and an exception occurs while the guest trace state lock is acquired (e.g. no more memory left to allocate a trace event on the heap), the guest tries to report to the host what the error was by calling the `out32` function. This function reports the guest trace data by serializing it on the spot, which needs to acquire the lock again, this leads to a deadlock. Signed-off-by: Doru Blânzeanu <dblnz@pm.me>
Signed-off-by: Doru Blânzeanu <dblnz@pm.me>
dblnz
force-pushed
the
fix-trace-bug-1
branch
from
8045903 to
2c69974
Compare
ludfjig
approved these changes
Dec 3, 2025
jsturtevant
approved these changes
Dec 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partially addresses #1032.
This fixes an issue where the guest hangs because of a deadlock produced when an exception occurs in the tracing code and the
GuestStatelock is acquired. When handling the exception, the call tooutbfunction, tries to acquire the same lock another time, which leads to a deadlock.