fix: resolve database deadlock: #4989
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `rotateWithLock` function holds a lock while it calls a callback function that's passed in by the caller. This is a problematic design that needs to be used very carefully. In this case, at least one caller passed in a callback that eventually relocks the mutex on the same thread, causing UB (a deadlock was observed). The caller was from SHAMapStoreImpl, and it called `clearCaches`. This `clearCaches` can potentially call `fetchNodeObject`, which tried to relock the mutex. This patch resolves the issue by changing the mutex type to a `recursive_mutex`. Ideally, the code should be rewritten so it doesn't hold the mutex during the callback and the mutex should be changed back to a regular mutex.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #4989 +/- ##
=======================================
Coverage 70.9% 71.0%
=======================================
Files 796 796
Lines 66727 66727
Branches 10981 10973 -8
=======================================
+ Hits 47333 47345 +12
+ Misses 19394 19382 -12
... and 3 files with indirect coverage changes |
scottschurr
approved these changes
Apr 15, 2024
HowardHinnant
approved these changes
Apr 15, 2024
seelabs
added
the
Passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
rotateWithLockfunction holds a lock while it calls a callback function that's passed in by the caller. This is a problematic design that needs to be used very carefully. In this case, at least one caller passed in a callback that eventually relocks the mutex on the same thread, causing UB (a deadlock was observed). The caller was from SHAMapStoreImpl, and it calledclearCaches. ThisclearCachescan potentially callfetchNodeObject, which tried to relock the mutex.This patch resolves the issue by changing the mutex type to a
recursive_mutex. Ideally, the code should be rewritten so it doesn't hold the mutex during the callback and the mutex should be changed back to a regular mutex.Type of Change