feat: concurrent download blocks on ibd #1957
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
driftluo
requested review from
doitian,
TheWaWaR,
keroro520,
zhangsoledad and
a team
quake
requested changes
Mar 7, 2020
doitian
requested changes
Mar 9, 2020
TheWaWaR
reviewed
Mar 9, 2020
driftluo
force-pushed
the
optimize-ibd-sync
branch
from
76cf3e1
to
8e0cf5b
Compare
TheWaWaR
reviewed
Mar 9, 2020
doitian
added a commit
that referenced
this pull request
Mar 10, 2020
feat: concurrent download blocks on ibd
doitian
requested changes
Mar 12, 2020
| pub fn try_update_best_known_with_unknown_header_list(&self, pi: PeerIndex) { | ||
| // header list Is an ordered list, sorted from highest to lowest, | ||
| // when header hash unknown, break loop is ok | ||
| while let Some(hash) = self.peers().take_unknown_last(pi) { |
driftluo
force-pushed
the
optimize-ibd-sync
branch
from
8e0cf5b
to
0e66df9
Compare
doitian
previously approved these changes
Mar 19, 2020
|
Hey folks, please review @quake @keroro520 @zhangsoledad |
quake
previously approved these changes
Mar 19, 2020
CKB - Pull Requests
automation
moved this from 👀 Awaiting review
to ✅ Reviewer approved
Mar 19, 2020
|
Please rebase |
doitian
added
the
s:waiting-on-author
driftluo
force-pushed
the
optimize-ibd-sync
branch
from
0e66df9
to
27f38fa
Compare
CKB - Pull Requests
automation
moved this from ✅ Reviewer approved
to 👀 Awaiting review
Mar 19, 2020
doitian
approved these changes
Mar 19, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As the chain height rises gradually, the synchronization duration increases linearly. This PR is the beginning of improving the synchronization time.
ckb synchronization is roughly divided into two blocks:
In the ibd process, currently only request a header map from a random node, and then initiate a download block request to the node based on the node's best-known header and the last common block calculated by the current tip.
In principle, block download does not limit the number of nodes requested, but because the ibd process only interacts with a node's header map during the ibd process, currently ibd can only download blocks from one node.
This pr uses the
headers_locator_hash_listin thegetheaderrequest from the other node to try to restore the best-known header of the node through the locally synchronized header map and then expands the number of downloadable block nodes from one to the current outbound numberI used a Hong Kong cloud host for testing, its configuration is:
before：
Sync from 0 to 1013645, took 184 minutes, average speed 5509/min
Sync from 0 to 1017469，took 148 minutes，average speed 6874/min
after:
8 outbound peer, sync from 0 to 1015441，took 72 minutes，average speed 14103/min
8 outbound peer, sync from 0 to 1017339，took 102 minutes，average speed 9974/min
8 outbound peer, sync from 0 to 1017339，took 100 minutes，average speed 10173/min
16 outbound peer, sync from 0 to 1023994，took 93 minutes，average speed 11010/min
32 outbound peer, sync from 0 to 494467，took 110 minutes，average speed 4495/min
When the number of sync nodes reaches 32, it seems that the download logic itself is a bit problem and further observation is needed, but this does not affect the modification of pr itself.