Synchronization optimization
The various messages used in ckb synchronization and the purpose of each message are defined in RFC, but it is only an outline for guidance. In concrete implementation, there may be various trade-offs that lead to slight differences in the implementation. This wiki is used to record the problems encountered in the implementation, solutions, proposed changes, etc.
Related PRs：
- https://github.com/nervosnetwork/ckb/pull/1979
- https://github.com/nervosnetwork/ckb/pull/1959
- https://github.com/nervosnetwork/ckb/pull/1985
The header map is a skip map with header hash as key and header + option point as value, which is used to quickly complete a traversal query to find the target header structure. This structure is a hotspot during synchronization, and by fixing this issue, the access time consumption can be greatly reduced.
Related PRs：
The last common header marks the smallest common block between this node and the other node, which is used as the start of synchronization to request a block from the other node, and the wrong header marker will lead to a huge amount of invalid lookups and downloads. This, combined with the redundant design that existed at the time, caused the synchronization to get stuck.
Related PRs：
During the synchronization process, if you need to download a block from a remote node, the problem to solve is how to know that the other party's best-known header is on the main chain of your own synchronization header while being higher than your own local tip header. As long as this problem is solved, any node can be a download source.
The IBD(Initial Block Download) process is silently processing a lot of messages. Now we just need to save the locator in the get_headers message sent by the other side, so that we can find the best-known header of the other side based on the local header map and enable the parallel downloading capability.
Related PRs：
- https://github.com/nervosnetwork/ckb/pull/1966
- https://github.com/nervosnetwork/ckb/pull/3261
- https://github.com/nervosnetwork/ckb/pull/3703
locator is an algorithm for exponentially sampling the chain of nodes, and its original algorithm for sampling is:
[latest 10 blocks，index - 2^n(n=1,2,3...)... genesis block]
In order to make the multi-node download open as fast as possible and keep it executing the whole time, a slight modification was made to the locator algorithm:
[latest 10 blocks，index - 2^n(n=1,2,3...)...**index >> 1**...genesis block]
At sampled index < 2 * step, the algorithm converts to a right shift of index, allowing the algorithm to include the entire chain in as uniform a manner as possible.
Related PRs：
The efficiency of ckb execution validation is not only the bottleneck of synchronization performance but also the bottleneck of TPS of the chain itself. Before, no matter what cell is relied on for execution, it needs to be temporarily loaded from the database and then executed. The above PR caches system cells, which are commonly used dependencies, in memory and only need to be read in memory during execution, thus significantly reducing the io overhead and improving the overall performance of the verification process by about 15%. The benefit scenario for this change is global.
Related PRs：
- https://github.com/nervosnetwork/ckb/pull/1999
- https://github.com/nervosnetwork/ckb/pull/2067
- https://github.com/nervosnetwork/ckb/pull/2628
- https://github.com/nervosnetwork/ckb/pull/3478
Several versions of the download scheduler have been implemented to optimize the distribution of download tasks, reduce duplicate downloads, improve node utilization, retire nodes that cannot provide service or are of low quality, and make the scheduler as adaptive as possible to various user environments. A little algorithmic optimization was done for some normal checks.
Related PRs：
Related PRs：
Reduced output transmission consumption helps improve performance
Related PRs：
During the synchronization process, a 24-hour-old block hash is specified, and all transactions prior to the synchronization to this block are skipped for validation, which is equivalent to a significant improvement in validation efficiency.