Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make File Operations async. (ArchivalState, etc) #75

Closed
dan-da opened this issue Dec 17, 2023 · 2 comments
Closed

Make File Operations async. (ArchivalState, etc) #75

dan-da opened this issue Dec 17, 2023 · 2 comments

Comments

@dan-da
Copy link
Collaborator

dan-da commented Dec 17, 2023

Related to #74.

Presently many, if not all, file operations use synchronous APIs, called from async functions. For example async fn get_latest_block() eventually calls get_block_from_record which opens a file and reads it in via (unsafe) mmap calls.

These sync operations will block the tokio executor which can result in poor performance/concurrency. So in general, it should be more performant to convert the file ops to async counterparts provided by tokio, or at least wrap them with spawn_blocking.

note: I'm unsure about making the mmap calls async; an area for research.

dan-da added a commit that referenced this issue Feb 18, 2024
Addresses #75.

Makes file operations in archival_state use non-blocking tokio APIs
instead of the blocking APIs in `std`.

Wraps mmap writes and reads in tokio::task::spawn_blocking(), to
make it async-friendly.
dan-da added a commit that referenced this issue Feb 18, 2024
Addresses #75.

Changes store_utxo_ms_recovery_data() and read_utxo_ms_recovery_data()
to use non-blocking I/O via tokio APIs.
@dan-da
Copy link
Collaborator Author

dan-da commented Feb 21, 2024

The only remaining blocking I/O that I'm aware of is in twenty-first storage. To fix that, we need to add a tokio dep in twenty-first, or move storage out of twenty-first somehow.

It's also possible there is blocking I/O in triton_vm or tasm-lib, but I'm hoping not.... ;-) Another consideration there is that compute-heavy functions should also be made into tokio tasks to avoid concurrency problems. I haven't investigated if we are doing that or not, eg when exec'ing VM code.

@dan-da
Copy link
Collaborator Author

dan-da commented Apr 6, 2024

closed by #124

@dan-da dan-da closed this as completed Apr 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant