memory-store: release locks earlier to avoid deadlocks #3668
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've been debugging a cause of flakey complement-crypto tests for about a month now. I was pretty convinced it was deadlocking somewhere in the memory store `save_changes` code. With additional logging, it's now clear that the there is an ABBA style deadlock when `save_changes` is called at the same time as `get_state_events`. I've also adjusted code for `get_user_ids` as it has a very similar pattern and also acquires locks in reverse order to `save_changes`, so is potentially vulnerable to this.
poljar
approved these changes
Jul 9, 2024
kegsay
added a commit
to matrix-org/complement-crypto
that referenced
this pull request
Jul 9, 2024
Previously we would only use persistence when the test strictly needed it e.g to test what happens between restarts. It's preferable to always run with persistence because: - it more accurately models real Element X clients e.g performance characteristics, runtime code. - any bugs in the DB layer are then also bugs in the client. Using the memory store has proven [error prone](matrix-org/matrix-rust-sdk#3668) and fixing these bugs don't improve the stability of EX at all. `ClientCreationOpts` will still have the `Persistence` flag though, as it remains useful to know when a test _needs_ persistence vs when it does not. In the future, we may clean up locally stored files during test runtime, and this flag would allow us to know whether it is safe to delete the files or not.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #3668 +/- ##
=======================================
Coverage 84.27% 84.28%
=======================================
Files 259 259
Lines 26662 26662
=======================================
+ Hits 22469 22471 +2
+ Misses 4193 4191 -2 ☔ View full report in Codecov by Sentry. |
This was referenced Jul 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've been debugging a cause of flakey complement-crypto tests for about a month now. I was pretty convinced it was deadlocking somewhere in the memory store
save_changescode. With additional logging, it's now clear that the there is an ABBA style deadlock whensave_changesis called at the same time asget_state_events.I've also adjusted code for
get_user_idsas it has a very similar pattern and also acquires locks in reverse order tosave_changes, so is potentially vulnerable to this.