Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize scheduler #2067

Merged
merged 3 commits into from
Jun 19, 2020
Merged

Optimize scheduler #2067

merged 3 commits into from
Jun 19, 2020

Conversation

driftluo
Copy link
Collaborator

@driftluo driftluo commented May 12, 2020 •
edited
Loading

Problems with the current scheduler:

  1. The calculation is too frequent
  2. Inability to adapt to complex network environments, e.g., increased network latency as the number of nodes increases, network latency exceeding expectations, etc.

This PR implements an adaptive scheduler based on past data, removing most of the redundant calculations. Translating the overall data into 4 intervals:

| fast | normal | penalty | double penalty |

The dividing line is the 1/3 position, 4/5 position, 1/10 position.

There is a 14/30 normal area, 1/10 penalty area, 1/10 double penalty area, 1/3 accelerated reward area.

Replace the 30-second clear-out strategy with a doubling penalty.

10%-20% increase in sync performance from testing, from 220-240 block/s to 250-270 block/s on a dual-core machine

the simulation test: https://github.com/nervosnetwork/sync-scheduler-simulation-test

@driftluo
Copy link
Collaborator Author

benchmark

@nervos-bot-user
Copy link
Collaborator

Benchmark Result

  • TPS: 359.98
  • Samples Count: 51
  • CKB Version: 70b6265
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo
Copy link
Collaborator Author

benchmark

@nervos-bot-user
Copy link
Collaborator

Benchmark Result

  • TPS: 294.03
  • Samples Count: 51
  • CKB Version: 25e539c
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo
Copy link
Collaborator Author

benchmark

@nervos-bot-user
Copy link
Collaborator

Benchmark Result

  • TPS: 354.71
  • Samples Count: 50
  • CKB Version: 70b6265
  • Instance Type: c5.xlarge
  • Instances Count: 3
  • Bench Type: 2in2out
  • CKB Logger Filter: info,ckb=debug

@driftluo driftluo marked this pull request as ready for review May 12, 2020 05:05
@driftluo driftluo requested review from a team, zhangsoledad, doitian and quake May 12, 2020 05:05
quake
quake requested changes May 18, 2020
View reviewed changes
sync/src/synchronizer/mod.rs Show resolved Hide resolved
sync/src/types.rs Show resolved Hide resolved
sync/src/types.rs Outdated Show resolved Hide resolved
sync/src/types.rs Outdated Show resolved Hide resolved
@driftluo driftluo force-pushed the optimize-scheduler branch 3 times, most recently from 16bf17d to 4790ced Compare May 21, 2020 05:26
@doitian
Copy link
Member

doitian commented Jun 12, 2020

@quake plans to give a better distribution models in simulation test to find better parameters after @keroro520 has collected new metrics. Before that we can merge this PR when there's no problems on the algorithm itself.

quake
quake previously approved these changes Jun 12, 2020
View reviewed changes
doitian
doitian requested changes Jun 18, 2020
View reviewed changes
sync/src/lib.rs Outdated Show resolved Hide resolved
sync/src/types.rs Show resolved Hide resolved
sync/src/types.rs Show resolved Hide resolved
sync/src/types.rs Outdated Show resolved Hide resolved
sync/src/types.rs Outdated Show resolved Hide resolved
sync/src/types.rs Show resolved Hide resolved
@driftluo driftluo requested a review from keroro520 as a code owner June 18, 2020 10:31
@driftluo driftluo requested a review from doitian June 18, 2020 10:31
doitian
doitian requested changes Jun 19, 2020
View reviewed changes
sync/src/types.rs Show resolved Hide resolved
@driftluo driftluo requested a review from doitian June 19, 2020 02:40
doitian
doitian previously approved these changes Jun 19, 2020
View reviewed changes
@driftluo driftluo requested a review from doitian June 19, 2020 09:52
@doitian
Copy link
Member

doitian commented Jun 19, 2020

Sorry that I misunderstood #2067 (comment) , the original code looks good.

@doitian
Copy link
Member

doitian commented Jun 19, 2020

bors r=quake,doitian

@bors
Copy link
Contributor

bors bot commented Jun 19, 2020

Build succeeded:

@bors bors bot merged commit 8b4af64 into nervosnetwork:develop Jun 19, 2020
@driftluo driftluo deleted the optimize-scheduler branch June 19, 2020 12:12
@doitian
Copy link
Member

doitian commented Jul 20, 2020 •
edited
Loading

The latest test result. The unit is block counts per second, higher is better.

  avg median min max count
eu_33_80_90 696 704 454 798 200
eu_50_75_90 674 683 483 786 193
eu_33_75_90 670 683 464 793 192
ap_33_80_90 662 666 475 759 190
ap_50_75_90 648 648 475 764 186
ap_33_75_90 636 650 474 720 182

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@quake quake quake approved these changes

@doitian doitian doitian approved these changes

@zhangsoledad zhangsoledad Awaiting requested review from zhangsoledad

@keroro520 keroro520 Awaiting requested review from keroro520

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@driftluo @nervos-bot-user @doitian @quake