Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Init sync arbitrary batch length + full bandwidth utilization #5978

Merged
merged 22 commits into from May 27, 2020

Conversation

farazdagi
Copy link
Contributor

@farazdagi farazdagi commented May 25, 2020

What type of PR is this?

Refactoring+Optimization

What does this PR do? Why is it needed?

  • Allows to utilize bandwidth efficiently by switching from epoch-based queue (where each epoch was assigned a FSM), to batches of arbitrary length (currently 64 - to match req/sec params).
  • Removes redundant code (in FSM mostly).

Relevant PR

Other notes for review

  • part of Init sync FSM, weighted RR, full-bandwidth optimizations #5798
  • This is almost the final PR within the current optimization round (concluding PR will have more tests to increase coverage - no functionality will be added). So, with --dev flag, when syncing from genesis, you'll will occasionally see around 100blks/sec. Overall, speed of syncing is ~70-80blks/sec.
  • Less requests needed to sync the same amount of chain blocks (previously we were restricted to 32 blocks/req, now it is arbitrary, and set to 64blk/req, so to sync the same chain we need half as many requests as previously)
  • Since core underlying data structure is changed in both queue and FSMs, I've tried to simplify and improve existing code benefiting from the knowledge about syncing I possess now (re-visiting init-sync was quite a fun project).
  • During last couple of days, I've synced dozens of times (with and without the --dev flag).

@prysmaticlabs prysmaticlabs deleted a comment from codecov bot May 26, 2020
@farazdagi farazdagi marked this pull request as ready for review May 26, 2020 13:08
@farazdagi farazdagi requested a review from a team as a code owner May 26, 2020 13:08
@farazdagi farazdagi added the Ready For Review A pull request ready for code review label May 26, 2020
@codecov
Copy link

codecov bot commented May 26, 2020

Codecov Report

Merging #5978 into master will decrease coverage by 0.34%.
The diff coverage is 82.38%.

@@            Coverage Diff             @@
##           master    #5978      +/-   ##
==========================================
- Coverage   59.89%   59.54%   -0.35%     
==========================================
  Files         321      320       -1     
  Lines       27632    26991     -641     
==========================================
- Hits        16549    16073     -476     
+ Misses       8866     8731     -135     
+ Partials     2217     2187      -30     

@prylabs-bulldozer prylabs-bulldozer bot merged commit f477804 into master May 27, 2020
@delete-merged-branch delete-merged-branch bot deleted the init-sync-arbitrary-batch-len branch May 27, 2020 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Ready For Review A pull request ready for code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants