Increase BLOCKS_METADATA_BUFFER_SIZE
to match (or be closer to) download batch size
#1364
Milestone
BLOCKS_METADATA_BUFFER_SIZE
to match (or be closer to) download batch size
#1364
The block downloader writes the blocks to disk and caches their metadata. We then flush the block metadata buffer to the Rust side to make the cache aware of the downloaded blocks.
Currently we flush the buffer when we reach 10 blocks of metadata. For the current download batch size of 100, this means we call
RustBackend.writeBlockMetadata
10 times per batch. Rust FFI calls are globally synchronized though thezc-io
thread, and the buffer flushes block the writes to disk inFileCompactBlockRepository.write()
. This means that if a download batch is running concurrently with the scanning of an earlier batch (as can happen due to buffering inCompactBlockProcessor.runSyncingAndEnhancingOnRange()
), the network I/O blocks on the Rust layer scanning.Instead, we should flush the buffer once the entire batch is written to disk. There is currently no benefit to early flushing (as we don't start scanning until the entire batch is downloaded), and if the download fails and causes the synchronizer to stop, then the un-flushed buffer doesn't matter because on restart all downloaded blocks are cleared.
Alternatively, we could decouple the network I/O from the
RustBackend.writeBlockMetadata
call, so that blocks continue to be downloaded and written to disk during the Rust FFI call.The text was updated successfully, but these errors were encountered: