-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make RustyLevelDB API async. #74
Comments
I've created a For now this lives in neptune-core, on a branch in my forked repo. But that's probably not where it belongs, because... This async issue also affects Currently, So, in order to make the DB access fully async, we will have to either add a tokio dep in
|
Closed by fa89d04 note however that the LevelDB iterator(s) remain sync, so any iterator callers need to do their own async wrapping using eg spawn_blocking() or block_in_place(). |
closing. |
LevelDB provides a synchronous API.
RustyLevelDB is a thin wrapper around LevelDB API. It also provides sync methods.
In neptune-core, we have async API like
get_block()
,get_block_header()
, etc that appear to be async. However internally they are calling these blocking DB APIs.This creates a hidden source of blocking/contention that may not be obvious until operating under load, and suddenly performance becomes very bad one day. See this description.
note: The RustyLevelDb is presently wrapped in a tokio lock, which requires await to acquire the lock. This gives an appearance of async, however the actual DB calls such as
get()
,set()
,delete()
remain blocking calls and may potentially block the calling task for database disk I/O.note: I'm in the process of removing this tokio lock, which is no longer necessary because the C++ leveldb does its own locking. While refactoring, this issue made itself clear.
A simple solution could be to make the RustyLevelDb methods async. Internally, they would wrap the DB calls inside spawn_blocking. This enables the RustyLevelDb method to await the DB operation (and return a Future immediately).
The text was updated successfully, but these errors were encountered: