Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase BLOCKS_METADATA_BUFFER_SIZE to match (or be closer to) download batch size #1364

Open
str4d opened this issue Jan 23, 2024 · 0 comments

Comments

@str4d
Copy link
Contributor

str4d commented Jan 23, 2024

The block downloader writes the blocks to disk and caches their metadata. We then flush the block metadata buffer to the Rust side to make the cache aware of the downloaded blocks.

Currently we flush the buffer when we reach 10 blocks of metadata. For the current download batch size of 100, this means we call RustBackend.writeBlockMetadata 10 times per batch. Rust FFI calls are globally synchronized though the zc-io thread, and the buffer flushes block the writes to disk in FileCompactBlockRepository.write(). This means that if a download batch is running concurrently with the scanning of an earlier batch (as can happen due to buffering in CompactBlockProcessor.runSyncingAndEnhancingOnRange()), the network I/O blocks on the Rust layer scanning.

Instead, we should flush the buffer once the entire batch is written to disk. There is currently no benefit to early flushing (as we don't start scanning until the entire batch is downloaded), and if the download fails and causes the synchronizer to stop, then the un-flushed buffer doesn't matter because on restart all downloaded blocks are cleared.

Alternatively, we could decouple the network I/O from the RustBackend.writeBlockMetadata call, so that blocks continue to be downloaded and written to disk during the Rust FFI call.

@str4d str4d modified the milestones: Zashi 1.0 performance, Zeboot Jan 23, 2024
@str4d str4d modified the milestones: Zeboot, Zashi 1.0 performance Feb 2, 2024
@true-jared true-jared removed this from the Android Zashi 1.0 milestone Apr 8, 2024
@nuttycom nuttycom added this to the Android Zashi 1.1 milestone Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants