Limit block import queue by segments, not blocks #2837
Labels
improvement
it is already working, but can be better
node
Node (service library/node app)
performance
Related to performance measurement or improvement
Milestone
One of the architectural decisions that was carried over from Substrate sync to our DSN sync is to limit import queue size in blocks. However, blocks can be of different size. Sometimes we have more blocks in a segment than the limit, sometimes less, but the goal is to limit memory usage.
As such I think we should track which blocks segments end at and download one or even more segments ahead of time, regardless of how many blocks are in there. It is actually nice that we deal with segments here due to their predictabe size, we can even try to download next segment while previous segment is still downloading if we have available concurrency to do that (for example if there is a few slow to download pieces left in the previous segment, we should be able to download the next segment already).
This will accelerate sync from DSN, though how much will depend on block size and how heavy blocks are.
The text was updated successfully, but these errors were encountered: