fix(server): use spawn_blocking when moving from async to sync contexts to prevent threadool deadlock #235
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
ysimonson
reviewed
Feb 10, 2022
ysimonson
reviewed
Feb 10, 2022
jasongoodwin
force-pushed
the
spawn_blocking
branch
from
543690e
to
2b96d38
Compare
…ts to prevent threadool deadlock
jasongoodwin
force-pushed
the
spawn_blocking
branch
from
2b96d38
to
9b7ecb6
Compare
jasongoodwin
changed the title
|
Checks pass. Bench on this branch vs master looks okay despite the small overhead of spawning extra resources into the secondary execution pool. Might be a small cost in the overhead but it's not large and a few runs I'm sure would show a pretty small delta in the average. I suspect ditching the mpsc would yield a better bench on this code. Will try it out. this: master: |
|
Was able to simplify/improve this a good deal. Pushed a change. Removes any dependency changes as well (preserve lock file etc.) This pr: master: |
jasongoodwin
force-pushed
the
spawn_blocking
branch
from
1c40e2a
to
de1e0d3
Compare
|
The pr a bit. This is good to go IMO. I've tested this and I believe very strongly that this is the right thing to do for consumers of the api. An interesting quality in the performance is that there seem to be more consist response times (lower standard deviation). |
ysimonson
reviewed
Feb 13, 2022
| @@ -44,6 +44,10 @@ fn map_conversion_result<T>(res: Result<T, crate::ConversionError>) -> Result<T, | |||
| res.map_err(|err| Status::invalid_argument(format!("{}", err))) | |||
| } | |||
|
|
|||
| fn map_tokio_result<T>(res: Result<T, tokio::task::JoinError>) -> Result<T, Status> { | |||
| res.map_err(|err| Status::internal(format!("{}", err))) | |||
| } | |||
There was a problem hiding this comment.
It looks like this is always called along with map_indradb_result? As a result, this could be simplified so that one function handles both.
There was a problem hiding this comment.
nice I'll do that.
There was a problem hiding this comment.
K pushed a small refactor for that. I chose to keep two methods and just unwrap one into the other as the inner portion is used in send.
There was a problem hiding this comment.
87ab385 - lmk if that looks okay.
|
Thanks for this fix! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue resolved here: the Datastore's methods are sync calls from async context w/ tokio runtime.
These should be called w/ block_on, not sync, as there is IO and you want to give the option to the implementor to use async themselves. You need to spawn_blocking on these async/sync barriers!
Any place that you're handing off from the runtime into a synchronous that may potentially block, you need to throw that into spawn_blocking. If the consumers of the Datastore api would ever do any concurrent/async work that needs to block to return to the sync api, then the threadpool can deadlock (eg futures::executor::block_on) or tokio will otherwise panic (eg Handle::block_on).
This change will allow concurrency/asynchronous code to be executed within the Datastore without forcing async trait on the consumer of the api.
For context, see: tokio-rs/tokio#2376
#235 (comment)
Checks run, the benches look marginally worse due to the overhead I think, but i only ran em once and who knows.